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Key messages 



A Current national system-level performance measures are insufficient for producing and maintaining 
quality vocational education and training (VET) institutions. 

A The model of performance measures suggested for VET institutions identifies relevant indices across 
the three dimensions of inputs (institutional resources, staff and student characteristics), processes 
(for example, quality of decision-making, institutional climate and culture) and outputs/outcomes (for 
example, student and employer satisfaction). Because of the importance of processes in linking and 
mediating inputs and outcomes, the model gives priority to measures relating to processes (such as 
quality of decision-making and institutional climate and culture). 

A Institutional self-monitoring and self-evaluation are considered significant challenges in improving the 
effectiveness of the VET system. 
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Executive summary 



This report addresses institution-level monitoring and evaluation, which is seen as the next challenge for 
improving the effectiveness of the vocational education and training (VET) system. The report argues 
that monitoring and evaluation at the institution level are best implemented through self-monitoring and 
self-evaluation. A process such as this requires attention to appropriate indices or measured indicators of 
trends of institutional performance. However, these indices are not necessarily the same as those needed 
for system monitoring and evaluation. 

Current accountability requirements would appear to be insufficient for producing and maintaining quality 
institutions, since these are designed to service the needs of national system reporting and accountability 
rather than individual institutions. They are not especially appropriate — certainly not sufficient — for 
servicing the needs of institutions to manage better. 

A new system is needed whereby institutions assume responsibility for their own improvement on the 
basis of empirical evidence of their own health and effectiveness. It is important that institutions adopt 
their own methods of data collection tailored to their own goals, context, characteristics and planning 
strategies. This requires attention to both the constraints on institutional decision-making and the 
corresponding opportunities for action. Technical and further education (TAFE) institutions do not have 
the same level of autonomy as universities, and the level of autonomy varies between jurisdictions. 
Nevertheless, in all cases, there are many institutional practices that are controlled locally. 

The basis for developing institutional capacity for self-monitoring and self-evaluation already exists in the 
Australian Quality Training Framework. The framework establishes standards for judging the quality of 
the institution’s delivery and assessment systems as well as their client services and administrative systems, 
and institutions will have to be compliant with them. One aspect of the standards is the requirement to 
conduct an annual audit. Institutions should use this requirement to implement comprehensive 
monitoring and evaluation of the institutional components and programs. 

The existing national data collection is too narrow and too cumbersome to be of much benefit in 
institutional self-monitoring and self-evaluation. Some of the data may be relevant and potentially useful, 
particularly in the form of benchmarks, but there are issues of accessibility and immediacy which need to 
be overcome to enable these data to be readily useable for institution-level monitoring and evaluation. 

Data need to be collected and used within short timeframes to ensure their currency and relevance. 
Further, much of the data relevant to state or nation-wide systems are not direcdy applicable to individual 
institutions, or are too sparse for reliable conclusions to be drawn internally. Institution-level monitoring 
and evaluation requires reliable data on institutional sub-systems, departments, units and programs. In 
addition, monitoring and evaluation at institution level needs to attend to issues relevant to local 
communities and industries, and this requires tailoring of questions and methods to fit the context. 

Institutions need to build their own capacity for data collection and data analysis. This report offers a 
model of performance measures which could be used for this purpose. The model draws on background 
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theory and practice reviewed in the report and identifies a range of relevant indices across the three 
dimensions: inputs; processes; and outputs/outcomes. These indices will need to be supported by relevant 
data drawn from existing collections (for example, student, staffing and finance), but with additional 
measures, particularly those related to processes (for example, quality of decision-making, institutional 
climate and culture) and perceptions of outcomes (for example, student and employer satisfaction). 

The model gives greater priority to the importance of processes (that is, program characteristics, 
procedural characteristics) in linking and establishing relationships between inputs and outcomes (student 
achievements), and identifies areas where prioritised and targeted actions can be taken to improve the 
effectiveness of the VET institution. The proposed indices can be selectively used and supplemented to 
examine institutional and program performance at the local level. Investment of resources in institutional 
self-monitoring and self-evaluation systems and processes increases the capacity of the institution to base 
planning decisions on deliberate and relevant empirical information about local institutional performance. 

National bodies such as the Australian National Training Authority (ANTA) and the National Centre for 
Vocational Education Research (NCVER) should play a key role in assisting institutions to build their 
institutional capacity for self-monitoring and self-evaluation. These organisations are ideally placed to 
develop and provide guidelines, procedures, techniques, resources and advice, and to disseminate 
examples of good practice. Enhancing institutional capacity will require investment in development and 
implementation at an institutional level. Furthermore, institutions will need some assistance in 
determining how to develop and implement appropriate strategies and, in this context, new strategies 
should ideally build on existing practice, and emerge through the encouragement of new ideas. To build 
on current practice, consideration should be given to funding case studies of interesting institutional 
practice. To promote new ideas, consideration should be given to funding innovative projects with the 
potential to provide exemplary models for other institutions, and as the basis of benchmarking between 
institutions. 
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Introduction 



This project began with an interest in how existing approaches to monitoring performance in the 
vocational education and training (VET) system in Australia might be extended to include institution-level 
monitoring. The reason for this interest is that, while broad system monitoring serves the useful function 
of establishing a picture of the overall functioning and health of the system at state and national levels, it 
is relatively powerless in leveraging change and improvement. Deciding about change that will lead to 
system improvement requires more systematic information on causal relationships and the magnitude of 
those relationships. However, even if such relationships can be broadly determined, ultimately, system 
performance results from the collective decisions of the participants in the system. Of critical importance 
as a locus of these decisions is the individual institution. It is at the institution level that the overall health 
of the system is determined, through the matrix of decisions within the institution relating to program 
design and delivery, organisational responsiveness and engagement, and learning support and 
development. Therefore, it is important to ask how institution-level monitoring and evaluation might be 
encouraged and supported. 

These considerations led to the additional question of whether existing data collections might be 
amenable to greater use in the provision of institution-level information which could be helpful to 
institutions for self-evaluation and improvement. In particular, could the existing data collections yield 
more elaborate information about the factors that contribute to successful outcomes and on which 
institutions could therefore act? Further, in what ways might these data need to be supplemented by 
additional data to enable institutions to develop more complete understandings of themselves, and 
approach self-evaluation and improvement more confidently. 



Setting 

Over the past decade, the state governments and the Commonwealth Government have established a 
comprehensive set of objectives for the national VET system, together with key performance indicators 
to measure the progress of the system against those objectives (see Robinson 2000). In recent years, the 
primary focus of reporting has been on levels of growth in the systems (state and national) in terms of 
increased resources, increased activity and increased efficiency. 

More recently, policy settings have emphasised the importance of quality in each institution’s capacity to 
deliver effective programs. Standards for the registration of training providers have been strengthened to 
address issues such as management, delivery and assessment (through the Australian Quality Training 
Framework). A range of support programs has been directed at helping institutions to meet these 
requirements. These appear to be working satisfactorily. 

However, there has been little research into the factors in vocational education and training in Australia 
which contribute to successful outcomes at the institution level, nor into the data required to identify and 
assess those factors. Fortunately, there is an existing body of research into institutional functioning, 
including schools as well as other institutions, which can be drawn on and made relevant to vocational 
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education and training. This project set out to explore the implications of that research and to develop a 
comprehensive model for institutional evaluation and monitoring in the VET sector. 



Approach 

The following research questions were the focus of the project: 

A What are the appropriate indicators of the performance of VET institutions and what factors are 
most likely to contribute to successful outcomes against those indicators? 

A Can existing national VET data collections be used as data sources for this purpose? 

A What gaps in data exist and what are the options for the collection of the required data? 

The order of these questions is important. The development of a conceptual model within which data 
can be situated needs to precede the analysis of the data. Then the data can be fitted to the model to test 
its efficacy. To the extent that the data are incomplete, additional data would need to be collected. 

This approach differs from an approach where existing data are first analysed ‘to see what they can tell us’ 
in an effort to provide partial explanations of the factors influencing performance, followed by attempts 
to identify the additional data needed to provide more complete explanation. Such an approach is 
sometimes referred to as ‘data dredging’. Although on occasions useful in initial explorations of the 
characteristics of a set of data, such an approach capitalises on unstable characteristics of the data and 
can lead to over-interpretation of relationships. It is preferable to adopt a confirmatory approach where 
the data are used to test the appropriateness of an explanatory model. 

For this reason the aim of this project was articulated as being the development of a model applicable to 
VET institutions for identifying and assessing performance outcomes and factors contributing to those 
outcomes. Subsequently, the existing national VET data collections were to be examined to determine 
their usefulness in providing relevant data for this model, as well as to identify gaps in the data collections, 
indicating the need for supplementary data. Finally, consideration was to be given to options for collecting 
the supplementary data and for future evaluation of the model. 

The project was conceived as consisting of four overlapping stages: 

A literature review 

A model development 

A data identification and analysis 

A model refinement. 

The first two of these stages are represented in the second and third chapters of this report respectively. 
These chapters represent an initial approach to the research questions and point towards possible future 
developments. They are not at this stage definitive analyses of the issues; rather, they offer suggestions for 
further elaboration and evaluation. It is hoped that they will generate widespread consideration and 
debate, and will lead to an enhanced capacity of the VET system to seek continual improvement towards 
greater institutional health and productivity. 

The data identification and analysis stage was more problematic. Initially, this was to involve examination 
of the characteristics of the National Centre for Vocational Education Research (NCVER) national 
database and analysis of a sample of that database. The purpose was to see how much of the model 
could be mapped in the database, what components of the database might be extracted for use in 
institutional self-evaluations and whether any relationships in the data might suggest critical measures to 
include in any such evaluations. Protocols prevented access to the database and data identification and 
analysis could not be conducted in this way. Instead, some broad conclusions are drawn about the use of 
the NCVER national database in terms of the proposed model. The researchers and NCVER mutually 
agreed not to undertake the fourth stage as a result of the principal researcher moving to another 
organisation. 
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The research for this project was undertaken and reported in 2002. 



Current performance measurement and accountability 

In recent years, attention has focused on the establishment and use of key performance indicators as a 
basis for system-level monitoring and accountability. The development of these indicators has been 
strongly influenced by the broader funding, policy and regulatory arrangements flowing from the inter- 
governmental Australian National Training Authority (ANTA) Agreement. 



National objectives and key performance measures 

Two principal documents give effect to the ANTA Agreement and guide priority-setting in vocational 
education and training — a national strategy and annual national priorities. The ANTA Ministerial Council 
sets the annual national priorities for the forthcoming year and approves VET plans developed by each 
state and territory, including performance measures against each priority, agreed financial inputs and 
outputs, and projected levels and areas of training delivery. 

The priorities are based on a national strategy developed by ANTA and endorsed by the ANTA 
Ministerial Council. The 1998—2003 national strategy outlines five national objectives for vocational 
education and training and contains seven national VET key performance measures. 

The objectives in this national strategy are: 

A enhancing mobility in the labour market 

A equipping Australians for the world of work 

A achieving equitable outcomes in vocational education and training 

A maximising the value of public VET expenditure 

A increasing investment in training. 

The agreed national key performance measures are: 

1 . Skill outputs produced annually within the domain of formally recognised vocational education and 
training 

2. Stocks of VET skills against desired levels 

3. Employers’ views on the relevance of skills acquired through vocational education and training 

4. Student employment outcomes and prospects before and after participation in vocational education 
and training 

5. VET participation, outputs and outcomes achieved by client groups 

6. (Actual) public expenditure per publicly funded output 

7. (Actual) public expenditure per total recognised output. 

The national objectives and key performance measures reflect the central policy concerns for vocational 
education and training of the state governments and the Commonwealth Government since the late 
1980s. These objectives can be summarised as: 

A ensuring that the VET system is responsive to industry needs 

A improving participation rates in and outcomes for individuals in vocational education and training to 
boost national skill levels 

A raising participation and attainment levels for under-represented groups 
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A increasing government and non-government investment in vocational education and training 

A building a national training market where a diverse range of providers competes to offer nationally 
recognised outcomes. 

This policy agenda has been strongly influenced by the increasing role of the Commonwealth 
Government in an area traditionally the responsibility of the states and territories. That influence has, in 
effect, been ‘purchased’ through a substantial increase in Commonwealth expenditure on vocational 
education and training and implemented by cooperative processes between the state governments and the 
Commonwealth Government and industry groups. 

The ANTA Agreement of 1992 provided the basis upon which the Commonwealth could increase its 
recurrent funding commitment without a reduction in state effort. There were subsequent modifications 
to this agreement. The states agreed to maintain their financial inputs in 1993, to maintain their outputs 
between 1994—97, to increase their outputs through increased efficiency from 1998—2001, and to match 
further increases in Commonwealth outlays on a 1:2 basis for 2002—04. The agreement also formalised 
the cooperative arrangements between the state governments and the Commonwealth Government and, 
through an independent board, provide industry leadership in the development of the national system. 

The progressive development of the national training system, including national objectives and key 
performance measures, has required the development of a comprehensive and consistent national VET 
data system to support performance monitoring and reporting. Since 1991 the NCVER has collected data 
from a range of sources as indicators of system-wide VET sector performance as a function of its role in 
the management and collection of national VET statistics. A primary responsibility of NCVER has been 
the development of the Australian Vocational Education and Training Management Information 
Statistical Standard (AVETMISS) for national data collection. This standard enables the collection of 
consistent data from a diverse range of provider institutions across all states. NCVER is also responsible 
for the development and enforcement of protocols for data handling which provide for both fair and 
equitable use of the data and protection of the rights of data providers. 

NCVER data holdings are primarily derived from four sources: 

A National data collections 

A Employer Satisfaction Surveys 

A Student Outcome Surveys 

A Graduate Destinations Surveys. 

In addition to these sources, additional data are collected irregularly, either as part of NCVER’s active 
research program, or in response to specific needs arising from NCVER clients. These include data from 
projects such as: 

A New Apprenticeships Tracking Survey 

Training in Australia Survey: Performance Measures in Private Education and Training 
A Employer Satisfaction with Graduates Survey. 

A simple search of the statistical clearinghouse lists 177 surveys with NCVER involvement. While it is 
clear that the NCVER data collection activities are extensive in scope, any examination of the use of 
current NCVER data holdings as sources of evidence about individual institutional health should be 
initially restricted to routinely collected core data assembled under uniform standards. This is not to say 
that other potential indicators may not be available from other VET sources, but the task of identifying 
them from the wealth of available data is beyond the scope of the current study. 

The original Australian Vocational Education and Training Management Information Statistical Standard 
(AVETMISS) fields were developed against an initial set of national objectives for vocational education 
and training drafted by a working group of officials and industry representatives agreed by Ministers for 
Vocational Education Employment and Training in 1991. These objectives were progressively refined to 
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support the objectives of the ANTA Agreement, the national VET strategy and national priorities. Key 
performance measures were developed to measure progress in these areas. AVETMISS data are 
supplemented by data derived from surveys which assess student and employer satisfaction and which 
track graduate destinations. AVETMISS data are also used to assess state performance and compliance 
with the requirements of the ANTA Agreement as outlined above. 

State objectives and key performance measures 

Under the ANTA Agreement, the states and territories are required to develop annual VET plans. These 
are submitted to and endorsed by the ANTA Ministerial Council on the advice of the ANTA Board. The 
state VET plans reflect the states’ responses to the agreed annual national priorities and include indicative 
performance outcomes against each priority. The state VET plans may also include additional state- 
specific strategies and priorities. 

The states also use AVETMISS data supplemented by data from within their own systems for planning, 
resource allocation and performance monitoring at the state level and provider level. 

Institutional objectives and performance measures 

States and territories ‘contract’ with training providers to deliver the agreed outputs. These arrangements 
take the form of performance and funding agreements with publicly owned technical and further 
education (TAFE) institutes and formal contracts with private training providers. 

Performance and funding agreements with TAFE institutes in most jurisdictions reflect the third to the 
seventh national key performance measures. They may also reflect other specific requirements, such as 
compliance with relevant government policies, procedures and regulatory requirements. Contracts with 
private providers usually only specify the output being purchased, but may also require compliance with 
policy, procedure and regulations. 

Each institute could also establish its own objectives and performance measures to reflect their individual 
strategic and business planning. Some TAFE institutes already establish additional performance measures 
relevant to local community or industry needs or which act as more holistic measures on institutional 
performance. Others will be limited to the performance measures contained in their performance and 
funding agreement with the state department. 

Reporting 

Reporting against the annual national priorities, the objectives of the national strategy and the national key 
performance measures takes place through a three-volume report produced by ANTA. Volume one 
provides a national overview of activities in the preceding year based on the annual national priorities; 
volume two contains reports from individual jurisdictions on the same basis; and volume three provides 
more detailed reporting on a current and historical basis against the eight key performance measures. 

Volume three of the annual national report is prepared by ANTA in collaboration with the NCVER, 
based on data from the AVETMISS collection. Individual jurisdictions prepare annual reports for 
government and parliaments. These may reflect performance against national key performance measures 
but will also incorporate reporting against ‘whole of government’ and state-specific performance 
indicators. TAFE institutes also produce annual reports in a consistent format in accordance with 
departmental or parliamentary requirements. 

Summary 

Key performance measures in vocational education and training reflect systemic national and state policy 
objectives and accountability requirements. The AVETMISS data collection has been designed to ensure 
that public reporting on VET performance and the accountability requirements of the ANTA Agreement 
are undertaken using a consistent format. Although AVETMISS data incorporate enrolment and other 
data supplied by VET institutions, they have not been used as a primary source of information to support 
performance assessment and monitoring at the institution level. 
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The need for more effective assessment of VET 
institutional performance 

In recent years there has been an increasing policy focus on the quality of provision by and performance 
of individual VET institutions, particularly TAFE institutes. This focus has been driven by several factors: 

A The establishment of a ‘training market’ in which public and private providers compete for both 
public and private funding has required VET institutions to directly respond to client needs. A range 
of commercial imperatives and business practices at the provider rather than the system level has been 
introduced as part of the training market. 

A The Australian Recognition Framework, and subsequently the Australian Quality Training Framework, 
established national standards for the registration of training organisations. Self-assessments and 
external audits have identified a range of areas whereby training organisations have been required to 
improve their business processes and reporting systems in order to demonstrate compliance. This has 
led to state and national VET authorities focusing on institutional as well as system-level performance. 

A Programs have been introduced to assist institutions to achieve regulatory compliance and to take 
advantage of new opportunities arising from national training reform. In particular, a national staff 
development program, Framing the Future, established an awareness of issues related to the 
performance of VET institutions, resulting in a dialogue between institutions and communities of 
staff working in related areas on issues specifically relevant to institutional and working group 
performance. Framing the Future was subsequently broadened to encompass a wider range of 
initiatives, including change management, strategic management, policy engagement, networking 
research and information (Mitchell & Young 2001). A specific focus was a program focusing on the 
development of high-performing VET organisations (Mitchell 2002) 

A The financial pressures arising from the introduction of competition, the need to generate additional 
revenue, and the consequent development of efficiency targets have led to internal and external 
reviews of individual institutions. 

The devolution of governance and management has increased the professional interest of senior 
managers in VET institutions in issues related to institutional performance, including national and 
international performance benchmarking and the incorporation of management and organisational 
development strategies from the public and private sectors. 

A The Victorian and Queensland state governments reviewed the role of TAFE as a public provider 
(Noonan 2002). These reviews included measures to improve the financial positions of TAFE 
institutes and the improvement of the performance of individual institutes by reinvesting in key 
business areas, such as human resources and management. In Victoria, the review recognised that 
current measures of performance of TAFE institutes were too narrowly based, particularly in relation 
to their broader roles in the community and in meeting the emerging requirements of a knowledge 
economy (Noonan 2002). 

Clearly, interest is increasingly turning to appropriate measures of the performance of VET institutions 
and the means by which these measures can be assessed. This requires consideration of the availability, 
relevance and veracity of data to support performance measurement at the institution level. 

While some national key performance measures are relevant to the performance of individual institutions, 
they provide only a limited view of the role of VET institutions, particularly the broader community roles 
of the public TAFE institutions. System-level measures do not say much about the internal ‘health’ of 
VET institutions, or the factors most critical to their performance. A different approach is needed to deal 
with institution-level performance evaluation. This report points to some potentially useful, emerging 
directions. 
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Relevant theories, concepts 
and practices in institutional 
performance evaluation 



Performance measurement has become an important part of the thinking on system planning and review 
in organisations as part of a movement emphasising total quality management and system accountability 
for effectiveness and efficiency in the expenditure of public funds. Schofield and Dryen (1997) have 
offered a definition of performance measurement as ‘a systematic way of assessing, judging or evaluating 
performance’ (p.3) and a performance measure as ‘a policy-relevant tool or instrument such as a statistic 
or description that permits an assessment, judgement or evaluation to be made about the performance of 
an institution or system’ (p.3). 

Most discussion in Australia in relation to performance measures for vocational education and training 
has been concerned with state/ territory and national reporting systems. The concern has been with the 
consequences or effects of VET policy and practice — what these consequences are and how satisfactory 
they are. An excellent recent review was produced by Dumbrell (2000) which surveys the research on 
VET system outcomes. In the context of this project, the headings used to present the findings are 
interesting. These are: labour market outcomes; equity in VET; youth unemployment and VET; funding 
the training market; matching supply with demand; and VET in schools. All of these are system-level 
issues and the associated recommendations are mainly concerned with further research; that is, developing 
a better picture of system outcomes. While useful for developing understandings about the overall 
functioning of the VET system and for informing overall VET policy, a framework such as this is not 
particularly relevant to decision-making at the VET institute level. 

A distinction is made in the terminology on performance measures and indicators between outputs and 
outcomes, with outputs conceived as direct immediate consequences of training programs (such as course 
completion rates) and outcomes conceived as subsequent consequences (such as changed employment 
status and perceived employment advantages). The distinction is useful to some extent, since outcomes, in 
this sense, involve the effects of factors other than the training program itself (such as job market 
changes and fluctuations and matching supply or demand). Even so, the distinction can lead to sterile 
debates about proper classification when what matters is that a range of consequences be considered. 
Further, all outputs/ outcomes are time-dependent; that is, affected by when the measure is taken. Thus, 
completion rates will be different if aggregated over time rather than calculated at a particular point in 
time (since completion is not confined to a particular timeframe). Also, ultimate employment prospects 
can be different from immediate prospects, and retrospective opinions of the quality and relevance of the 
training program can change over time. 

The 1998—2003 key performance measures have specified data (or indicator) requirements which 
operationalise the measure and reveal more about their meaning. A summary of these is given below 
(ANTA 1998): 

A KPM 1 : Skill outputs produced annually within the domain of formally recognised VET 

♦ number of recognised qualifications achieved 

♦ number of modules successfully completed 

♦ number of units of competency attained. 
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A KPM 2: Stocks of VET skills against desired levels 

♦ total population completed qualifications, modules and units of competence (by industry and 
occupation) — from Australian Bureau of Statistics and by estimation from existing data and 
research 

♦ profile of industry demand for VET skills (by estimation). 

A KPM 3: Employers’ views on the relevance of skills acquired through VET 

♦ employer satisfaction measures through Employer Satisfaction Survey. 

A KPM 4: Student employment outcomes and prospects before and after VET participation 

♦ student change in employment status and satisfaction measures through Graduate Destination 
Survey and Student Outcomes Survey, the former by census every three years and sample every 
other year, and the latter by sample every three years. 

A KPM 5: VET participation, outputs and outcomes achieved by client groups 

♦ breakdowns of KPM1 and KPM2 by equity groups and industry types. 

A KPM 6: (Actual) public expenditure per publicly funded output 

♦ total government operating expenditure on vocational education and training 

♦ publicly funded outputs (from KPM 1). 

A KPM 7: (Actual) public expenditure per total recognised output 

♦ total government operating expenditure on VET 

♦ total recognised outputs (publicly and privately funded). 

Key performance measures 1—5 are said to be effectiveness measures and key performance measures 6—7, 
efficiency measures. In terms of the outputs/outcomes distinction, Dumbrell (2000) points out that 
number 1 is an output measure while 2—5 are outcomes measures, at least in part. However, what is 
striking about these key performance measures is both their lack of precise definition (in most cases 
relying on procedural interpretations or research methodologies which can affect the validity of the 
measures) and their purely descriptive nature — with an absence of any reference to the standards or 
benchmarks or methods of comparison which will enable their evaluative interpretation. That is, the key 
performance measures specify a database, but lack measurement integrity and an evaluative framework. 

As Dumbrell (2000) points out, the key performance measures may provide a useful database from which 
to evaluate the impact of broad policies, provided, we would say, the measurement integrity and evaluative 
framework issues are addressed. However, they are of limited utility for teachers and administrators in 
VET institutions. Further, it is unlikely that a disaggregation of the national database for each institution 
would provide the necessary data. The focus of the key performance measures is on the system level and 
they are insensitive to many of the issues which need to be addressed at institution level. Also, the 
timeframes for collection and analysis are too long to satisfy the needs of VET institutions for 
progressive monitoring and improvement in their institutional operation and program delivery. For 
sensitive and rapid response to institutional needs, a ‘bottom up’ approach is needed to building evaluative 
systems. Examples of thinking which can inform such a process exist and can be utilised. Some of these 
are examined in this chapter of the report. 

The following discussion is divided into two sections: research and development on performance 
measures and indicators relevant to the evaluation and improvement of VET institutions per se; and 
research and development on factors and indices relevant to evaluating and improving organisations in 
general. The former derives from reports and documents situated specifically within the VET sector, the 
latter from reports and documents concerned with organisations more generally — that is, organisations of 
any type — and educational organisations more generally, especially schools and colleges. These are 
categories of convenience rather than of principle. In fact, VET institutions can be usefully compared 
with organisations of any type, including schools. While it is important to recognise the particular 
characteristics of VET institutions which make them different from other organisations, it is also useful 
to identify the similarities. Further, much of the discussion of VET-specific performance measures itself 
indicates derivations from the more general organisational performance measurement research literature. 
However, one striking feature of the field is its eclecticism. There is no compelling overall model of 
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organisational performance to which the existing literature can be related. An initial attempt is made to 
move towards greater coherence in the following chapter of this report. 



VET performance measures and performance indicators 

General considerations 

VET sector services are delivered to students and providers via a number of institutional arrangements. 
While the context of each differs, often markedly, such institutions do share a number of common 
features as organisations. There are, however, limitations to a generic organisational approach. It is not 
possible to develop a generic list of indicators applicable in all situations. Some adaptation to the 
particular context and characteristics of the institution may be necessary. With specific reference to VET 
institutions, Van den Berghe (1998) points out that it is best to develop indicators that ‘... reflect 
particular quality characteristics or quality goals of the institute . . .’ (p.37). That is, it is necessary to 
conceptualise institutions as individual entities with their own goals and dynamics. This may indicate a 
need to develop indicators which are more comprehensive and adaptable, and which complement the 
more traditional systemic data collection. 

Some confusion and inconsistency exists about the meaning of the terms ‘performance indicator’ and 
‘performance measure’. Within the discourse of measurement theory, performance indicators can be 
qualitative as well as quantitative, but performance measures are necessarily quantitative. This is not 
universally accepted. Within the discourse on vocational education and training, these meanings would 
seem to be reversed. For example, van den Berghe (1998) sees indicators as essentially statistical (although 
they could involve yes/no categorisation as well as scaling, and subjective judgment as well as objective 
measurement). A similar subscription can be seen in Schofield and Dryen (1997) and MacGraw and 
Peoples (1996) as well as the documentation on key performance measures (ANTA 1998). The most 
consistent approach is to adopt the term ‘performance indicators’ as the generic term for both 
quantitative and qualitative data and this will be the approach taken here. 

In any case, a mix of quantitative and qualitative indicators is often recommended (MacGraw & Peoples 
1996). A persistent difficulty is that a ‘standard’ or ‘expectation’ is rarely specified, presumably because it 
would be somewhat arbitrary, which means that performance indicators and measures are largely 
descriptive, at least in the first instance. Only when they are interpreted in the light of other information, 
such as by comparison with other systems or other institutions or with the same system or institution on 
previous occasions, can meaning be attached to the data. Sometimes, however, the ‘standard’ is built into 
the data, such as when students or employers rate the quality of a program. In that case, the raters are 
ascribing their own desired or expected standard against which the program is rated (for example, when 
indicating whether a program was ‘satisfactory’). Even so, the aggregate of their ratings must itself be 
interpreted in terms of adequacy or excellence (what percentage of ‘unsatisfactory’ ratings, or average of 
ratings, before mild/ serious/ alarmed concern is registered?) In any interpretation of performance data, it 
is important to ask whose values and expectations are being represented and whether the interpretations 
of various stakeholder groups have been sought and considered. 

In an early report on performance indicators (Commonwealth Tertiary Education Commission 1989), it 
was suggested that indicators relating to teaching be divided into internal performance indicators (such as 
graduation rates), operating indicators (such as class sizes) and external indicators (such as employment 
destinations). This is similar to an inputs, processes and outputs /outcomes classification. However, the 
examples (graduation rates, class sizes and employment destinations) illustrate the absence of any in-built 
standard or benchmark for desirable performance. Desirable performance on such indicators is clearly 
related to the context — what may be desirable in one context may be inappropriate in another. Further, 
interpretation depends on the values and expectations of the interpreter. Consequently, performance 
indicator data can be viewed best as a basis for discussion and debate among the stakeholders rather than 
definitive requirements in themselves. That is, performance indicators can be seen as an analytical 
structure for capturing and communicating relevant information and revealing system characteristics for 
scrutiny (see Hattie 1990). 



16 



Managing better: Measuring institutional health and effectiveness in VET 




Australian research and development 

Performance indicators serving accountability 

Hattie (1990) argued that the move towards greater accountability makes performance indicators of one 
kind or another inevitable, but that it is important that they be able to capture the ‘delicate and critical fine 
strokes’ and not just paint the broad picture (p.250). He suggests that ‘while there is a need for a global 
view [involving gross overall indices], it must be balanced with detail’ to capture ‘the nuances of success 
and failure’ (p.251). Hattie proposes a model of performance indicators adapted from Shavelson (1988) 
which differentiates between inputs (institutional resources, staff characteristics, student characteristics), 
processes (program characteristics, procedural characteristics) and outcomes (student achievements). 1 He 
points out that this is a static or non-predictive model, lacking relationships between and weightings for 
the indicators (as would be required in a structural equation model). He also points out that it has been 
argued (Cullen 1987; Scriven 1988) that input and process indicators are not themselves performance 
indicators even though they are useful, even important, for explaining the outcomes. A further argument 
could be made that this is a rather narrow representation of outcomes since the use made of raw resource 
inputs and the experiences students have in their studies can themselves be seen as consequences flowing 
from decisions and resource deployment and program implementation, and are important in themselves 
(for example, in terms of staff and student morale), not just for their subsequent consequences. 

Guthrie (1991) gave an account of the initial development of thinking about performance indicators for 
the national VET system, dating important initiatives from around 1988. He suggested that indicators 
needed to be relevant to the mission statements and corporate goals which referred to planning, using and 
reporting, quality and quality improvement, accountability, efficiency and effectiveness, meeting needs and 
better management. He stressed that performance indicators should not be collected as ends in 
themselves, but as means to assist in achieving these goals. In this respect, he saw performance indicators 
as tools for enhancing the quality of the system through their contribution to accountability and 
management. At the time of Iris review, there was much variability among the states and territories. A 
useful taxonomy, more evident at the time in some states and territories than others, involved: 

A financial indicators (such as actual versus budgeted expenditure and cost per student hour) 

A student indicators (such as completion rates, student satisfaction, student destination) 

A access and equity indicators (enrolment and graduation rates for different social groups) 

A staff indicators (such as staff development statistics and attrition rates) 

A employer indicators (such as industry satisfaction) 

A commercial activity indicators (such as level of activity with industry in priority areas). 2 

With respect to measures of efficiency, Guthrie (1991) notes, citing a report by the Queensland 
Government department responsible for vocational education and training at that time (Bureau of 
Employment, Vocational and Further Education and Training 1990), that efficiency measures need to be 
interpreted in terms of their context. Specifically: 

A Comparisons should involve institutions with similar output profiles. 

A New, smaller and specialist institutions have higher operating costs. 

A Different fields of study have different operating costs. 

A Efficiency measures indicate only activity levels not quality levels. 



1 He also includes staff achievements (such as research publications) under outcomes since he is referring to universities. Staff achievements 
are less relevant in VET institutions. 

2 It is of some interest to note that Guthrie (1991) provides a substantial summary of the structures of the Queensland vocational education 
and training system at the time and the approach taken to program management, including both external reviews and institutional self- 
evaluation, the latter supported by a set of guidelines and a manual and deriving from Byrne, Houston and Thomson (1984a, 1984b). 
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Performance indicators serving improvement 

Guthrie (1991) notes that at the time of his report several instances had emerged of approaches to 
institutional self-audit and self-review which made use of performance indicators. No information is 
available about whether these were implemented and whether they were successful. No differentiation was 
made, however, between indicators appropriate for different levels of decision-making. Guthrie (1991) 
notes this as a confusion about whether indicators are a developmental tool or a measuring stick. He 
draws attention to the: 

. . . uneasy relationship between the use of indicators for development versus measurement 
and control, and for accountability versus self-regulation, performance monitoring and 
improvement. These tension are real, and reflect the balance between the strength of the 
political forces acting to control or to shape what an organisation does — its action and its 
priorities. It is, in a nutshell, the balance between external control (or a measure of it) and 
self-regulation. . . . However, if accountability is ‘bottom up’, conformity and consistency 
are the essential elements and imply commitment to comparison linked to external control. 

‘Top down’ accountability, on the other hand, is about diversity and customisation. 

Comparison in these circumstances is meaningless. It therefore depends on who is the 
focus of accountability — those ‘above’, or [those] ‘below’ who are served by the 
organisation (i.e. its customers or clients). (Guthrie 1991, p.40) 

Guthrie (1 99 1) saw signs of movement from accountability to improvement: ‘if properly conceived 
[performance indicators provide] a way of diagnosing illnesses before they become terminal’ (p.50). He 
also emphasised that performance indicators: 

dr are best seen as elements in a management information and quality improvement system 

dr have the potential to promote discussion and debate about program quality and program delivery 

dr should be chosen on the basis of importance and appropriateness 

dr should be chosen so that they foster rather than restrict diversity and good practice 

dr should be interpreted in context and in relationship with other indicators 

dr should involve a range of indicators to obtain a valid indication of performance 3 

dr should involve attention to the quality of the data collected 

dr should respect and encourage a sense of ownership and participation among staff. 

The theme of multiple measures, or a range of indicators, was taken up again by Coleman (1993) who 
stressed that multiple indicators are more robust than single indicators. Other points offered include: 

dr Both quantitative and qualitative indicators are useful. 

dr Participative development leads to ownership and use. 

dr Develop indicators with the intended use in mind. 

dr Balance the costs of collection against the potential benefits. 

MacGraw and Peoples (1996) developed and trialled a comprehensive system for collecting information 
on program delivery related to the teaching and learning process. They identified 1 5 ‘performance 
measures’ (actually qualitative information collected through interviews) grouped into five focus areas. 
Each focus area was defined by key ‘elements’ and each ‘performance measure’ by lead questions 
supplemented by prompts. Separate lead questions and prompts were developed for application to 
teachers/ trainers and students. The focus areas and associated ‘performance measures’ were: 



3 



In social research, this is referred to ‘triangulation’ which involves strengthening the validity of data by using a combination of sources and 
methods. 
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A Focus area 1 : Planning and preparation 

♦ Teacher/ trainer as researcher 

♦ Delivery structure and sequence 

♦ Organisational culture 

♦ Customer/ client expectations 

A Focus area 2: Content and relevance 

♦ Industry currency of teacher/ trainer 

♦ Industry relevance of material 

A Focus area 3: Delivery 

♦ Teaching/ training and learning approaches 

♦ Communicating with students 

♦ Management of teaching and learning 

A Focus area 4: Assessment and evaluation 

♦ Assessment methodology 

♦ Student learning from assessment 

♦ Teacher/ trainer evaluation 

A Focus area 5: Relationship between teacher/ trainer and student 

♦ Learning environment 

♦ The student as individual 

♦ Student diversity 

The research associated with the development of this schema showed that VET teachers /trainers were 
enthusiastic about using this approach for self-evaluation and that the developed schema could provide a 
useful basis for this. They recommended that it be developed into an instrument for use by individuals 
and teams as part of a process of regular self-assessment and continuous improvement in their teaching. 

Equity issues 

Equity considerations typically focus on participation and performance of particular social groups. The 
following groups are identified as equity groups within the Australian VET system (ANTA 1997): 

A women 

A Aboriginal and Torres Strait Islander peoples 
A people with a disability 

people from non-English speaking backgrounds 
A people without adequate literacy and numeracy skills 
A people from rural and isolated areas. 

The ANTA report (1997) points out that these groups are not homogeneous, in that some members of 
each group may not experience disadvantage, while some others may experience multiple disadvantage. 
Further, disadvantage is not limited to these groups. So, while data on these groups are important for 
gauging the responsiveness of the system to different needs, responsiveness should be more broadly 
conceived within the system to encompass the capacity to respond to all student needs. 

Two approaches to equity are apparent — social justice and managing diversity (ANTA 1997). Both of 
these approaches can be adopted and have complementary implications for monitoring system and 
institutional performance. The social justice approach focuses on removing impediments to participation 
and success in the wider society — such as economic, social, attitudinal, legislative and administrative 
factors beyond the individual’s control. The managing diversity approach focuses on creating and 
developing strategies responsive to different needs of students and which respect their different 
backgrounds, perspectives and skills: 
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Managing diversity calls on a system to reflect the diversity of its client base in its 
structures, personnel and employment practices and, in the vocational education and 
training context, to adjust teaching, training, learning and assessment to encompass 
difference. Development of an ethos of inclusiveness, respect for difference and the 
inclusion of people from a range of client groups in decision-making processes are 
major aspects of this approach. Managing diversity offers a positive response to 
changing workforce and population characteristics, without targeting particular groups 
or identifying specific areas of disadvantage. (ANTA 1997, p.5) 

ANTA (1997) suggests that a strategic approach to equity requires attention to three elements: 

A overcoming or removing structural inequities 

A implementing targeted responses to equity based on ‘workable solutions’ 

A introducing resource allocation strategies and incentives which encourage responsiveness to client 
needs 

These expanded notions of equity require that performance indicators extend beyond participation rates 
to include outputs/ outcomes (that is, attending to success not just to access) and also ‘the capacity of the 
system to respond to a diversity of clients’ (ANTA 1997, p.8). Since this requires attention to ‘flexibility 
and responsiveness at the point of delivery’ (ANTA 1997, p.8), the implication is that monitoring must 
occur at the level of the individual institution delivering the training. The strategies and incentives put in 
place within the institution to create flexibility and responsiveness at all levels of the institution in 
responding to individual student needs, both for special equity groups and more generally, should be 
monitored and evaluated for their appropriateness and successfulness. 

Schofield and Dryen (1997) support the need for an improvement in provider-level monitoring of equity 
provisions and outcomes. However, their research found little attention given to this in provider 
institutions, apart from gender breakdowns of participation statistics (p. 1 5) . They point to the need to 
include output/ outcome measures as well as participation measures in monitoring equity (as now included 
in key performance measure 4), but also to attend to process or activity information to evaluate the 
success of targeted strategies. They draw attention to the need for such data to establish whether there is 
a causal link between differential delivery strategies and outcomes; that is, whether there are gender- 
specific links between modes of delivery and outcomes. This can be seen as testing empirically the extent 
to which flexibility and responsiveness are essential for this equity group (and by implication for other 
equity groups) . 

Flexibility and responsiveness are two key principles sought for Indigenous students in vocational 
education and training in a report by the Student Services Committee (1997). Two further principles are 
accessibility and quality. All four principles were initially defined by the predecessor to this committee in 
an earlier report (Education and Student Services Standing Committee 1995). Both reports are sceptical 
of the applicability of system-level monitoring and evaluation and system-wide criteria for success in the 
improvement of vocational education and training for Indigenous people. The later report (Student 
Services Committee 1997) recommends the adoption of ‘a framework against which localised 
benchmarking can be developed based on the contextual needs of students and institutes’ (p.5). It also 
recommends that performance indicators for Indigenous education in vocational education and training 
should address ‘issues of self-determination, student support needs, and the range of learning settings’ 
(p.4). In other words, performance indicators should be contextually and culturally sensitive, a principle 
which would seem appropriate for all groups and settings, each unique in their own way. 

In applying these principles to the provision of VET services for Aboriginal and Torres Strait Islander 
students, the Students Service Committee (1997) considered the following potential performance 
indicators to be critically important and culturally relevant: 

4* the extent of self-determination and local community involvement in the development of 
programs 
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A the extent of equitable access, participation and outcomes for Aboriginal and Torres Strait 
Islander people 

“y" the extent of support for literacy and numeracy development 

the extent of support for Aboriginal and Torres Straight Islander people in urban, rural and 
remote communities 

■v" the extent of integrated service delivery and support in learning environments 

A the extent to which organisational structures support the provision of student services for 
Aboriginal and Torres Strait Islander people. (Student Services Committee 1997, p.49) 

The report also provides a more elaborated framework for the evaluation of student services for 
Aboriginal and Torres Strait Islander students under the four desirable principles (accessibility, 
responsiveness, flexibility and quality) and in relation to four characteristics (range of services, location, 
staff, integration). Table 1 (page 22) provides some edited extracts from this framework. Although the 
main focus is student support services, this framework provides some suggestive possibilities for the way 
in which institutional evaluations might proceed more generally. 



European research and development 

Van den Berghe (1998) undertook a major review of performance indicators in use in the vocational 
education sector in the European Union which is worth substantial mention here. His study addressed the 
following key questions: 

What are criteria for determining quality indicators for vocational education and training? 

What types of indicators already exist, and which types are lacking? When and how can 
quality indicators be used? What makes quality indicators for VET different from those for 
general education? Should indicators be developed preferably at institutional or at macro- 
level? (Van den Berghe 1998, p.ix) 

These key questions are all pertinent. Some important conclusions from Van den Berghe (1998) can be 
summarised. First, a useful two-dimensional model is offered for characterising performance indicators: a 
message dimension and a purpose dimension. The message dimension runs between the extremes of 
descriptive/static/input-oriented and quality/ dynamic/ output-oriented and has four main ‘types’: 

A descriptive indicators (such as frequencies and amounts) 

A management and policy indicators (such as trends and relationships) 

A output indices (such as dropout and completion rates) 

A quality indicators (such as teaching quality ratings). 

The purpose dimension runs between measurement/ analysis/ evaluation and normalisation/ standardisation 
and has three main ‘types’: 

df- problem identification (to identify desirable change) 

A communication (to announce trends and accomplishments) 

A comparison (against a benchmark or between institutions or programs). 4 

Typical indicators are an overlapping mix of types mapping an area in the two-dimensional array. This 
provides a basis for clarifying where particular performance indicators sit in terms of the range of 
messages and purposes. 



4 ‘Normalisation/ standardisation’ is van den Berghe’s (1998) term and may have European currency. The intention would seem to be 
norm referencing/ criterion referencing. 
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Table 1: 



Extracts from framework for evaluation of student services for Indigenous students 



Accessibility 

Range of services 

The range of student services and facilities is appropriate for the institute’s profile of Aboriginal and Torres Strait Islander 
students. 

Professional support services are readily accessible for Aboriginal and Torres Strait Islander students, including counselling, 
career, employment, library services. 

Location 

Aboriginal and Torres Strait Islander student utilisation of student service functions and satisfaction with services. 

Staff 

Appropriately qualified staff are identified and positioned in the institute. 

Integration 

Evidence of shared responsibility and integrated provision of support for Aboriginal and Torres Strait Islander students. 

Responsiveness 

Range of services 

Appropriate processes in place to gather client satisfaction and feedback from Aboriginal and Torres Strait Islander students. 

Barriers to study and access to support services for Aboriginal and Torres Strait Islander students are identified and acted 
upon. 

Location 

Strategies are in place to enable Aboriginal and Torres Strait Islander students of other campuses to access student services. 
Staff 

There is regular contact and information sharing between teaching/faculty staff and Aboriginal and Torres Strait Islander 
support staff as a mechanism for assessing and monitoring student needs. 

Integration 

Aboriginal and Torres Strait Islander support staff have access to committees [and have] feedback loop to senior management. 
Participation of Aboriginal and Torres Strait Islander support staff on institute committees. 

Flexibility 

Range of services 

There are well developed community networks established to meet any gaps in the provision of support for Aboriginal and 
Torres Strait Islander students. 

Staff 

Aboriginal and Torres Strait Islander support staff have access to negotiated professional development strategies which 
address their particular needs. 

Quality 

Range of services 

Evidence of strategies which have improved the number of Aboriginal and Torres Strait Islander students and the quality of 
participation of those students. 

Staff 

Staff recognise the particular needs of Aboriginal and Torres Strait Islander students and have access to cultural awareness 
training. 

(Student Services Committee 1997, pp.53-5) 

Van den Berghe (1998) issues two caveats concerning performance indicators: first, perfection is neither 
attainable nor desirable — it is important to see performance indicators as a tool, not as an end in 
themselves. Second, the effort expended on developing and implementing performance indicators should 
be commensurate with their importance and usefulness. 

Concerning the current interest in quality indicators in educational institutions, Van den Berghe (1998) 
observes: first, that while concern for quality is not new, it was previously characterised by a narrow focus 
on particular features of educational delivery; second, there is an increasing focus on the total 
effectiveness of the educational institution or training provider’s organisational capacity to deliver high- 
quality services according to the concepts of total quality management; third, this is part of a broader and 
permanent movement; and fourth, and importantly, this shift is being realised in different ways in 
different educational sectors, in particular, that the vocational education sector is characterised by: 
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A a focus on overall institutional performance 
A new or additional quality control mechanisms 
A quality assurance mechanisms. 

Despite this institutional and system focus, Van den Berghe (1998) warns that individual teacher/trainer 
performance is a crucial component of institutional success and that teacher commitment and proficiency 
must be supported if effective learning is to occur. In other words, an institutional focus does not ignore 
the constituent parts of institutional functioning. Rather, it situates those constituent parts within the 
context of overall institutional structures and mechanisms. 

While these theoretical considerations are quite convincing, Van den Berghe (1998) was unable to find any 
instance of their implementation. Rather, he found partial implementations which usefully illustrate 
various components of a complete system. One interesting example came from the Emilia-Romagna 
region in Italy which involved accreditation criteria for training organisations and training programs. Ten 
indicators on four priority criteria were rated on a scale of 1 to 5. The indicators are shown in table 2. As 
Van den Berghe points out, although interesting, these indicators are concerned only with inputs 
(preparation, not delivery or consequences) in relation to a program (not the whole institution) with a 
focus on normative comparison (not improvement), requiring subjective judgement by expert assessors 
(not involving other stakeholders). Also, these performance indicators are particular to the Italian VET 
system. Nevertheless, they point towards a style for representing performance indicators using rating 
scales. 

Table 2: VET indicators and quality criteria from the Emilia-Romagna region in Italy 

Criterion 1 : Relevance with regard to labour market needs and development projects 

Relevance in relation to sectoral, geographical or developmental needs 
Importance of the problems to be addressed 
Involvement of socio-economic actors. 

Criterion 2: Relevance with regard to the needs of potential users 

Adequacy and potential take-up with regard to the real needs of potential users. 

Criterion 3: Accuracy and adequacy of the overall training program 
Consistency and details of the training content in relation to the objectives 
Adequacy of the training approach (didactics, duration, assessment, ...) 

Consistency between training content and the organisation put into place (trainers, materials, ...). 

Criterion 4: Existence of a documented professional profile 
Comprehensiveness of the description of the professional situation targeted 
Detailed description of the competence profile 
Validation of the competence profile by employers’ representatives. 

Source: Van den Berghe (1998, p. 40) 

A broader system, PROZA (Project Group for Developing Self Analysis quality assurance instrument), 
although not as fully developed, is reported from a group of Flemish higher (mainly vocational) education 
institutes. The system is quite comprehensive, covering almost 90 quality criteria (grouped under nine 
headings), with five quality stages for each criterion and four or five questions per stage (giving a total of 
about 2000 questions). Various methods of data collection can be used and the focus is on institutional 
self-assessment, analysis and improvement. 

Another example given by Van den Berghe (1998) is from the Danish Department of Vocational 
Education and Training. Indicators pertaining to institutional operational quality are shown in table 3 and 
cover a range of qualities relating to inputs and processes. It is reported that quality criteria were under 
development. 
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Table 3: Components of the Danish strategy plan for VET relating to institutional operational quality 

Strategy development 

Management instruments 

Strategic management, including school profiling 

Adaptation strategies to changing goals and needs 

Local educational plans and curricular work 

Drawing-up of budgets 

Educational instruments 

Students' right to be consulted — participation 

School culture and environment 

External contacts 

Cooperation with local education and training committees 

Collaboration with other schools and colleges: locally, regionally and nationally 

International activities 

Resource parameters 

Allocation 

Planning of supply of courses and services 
Economic management and cash-flow control 
Staff recruiting and policy 

Equipment and physical facilities (including library) 

Registration of students’ throughput (including completion rates) 

Operational aspects 
Guidance, introduction 
Special educational assistance 
Safety/working environment 
Organisation of examinations 

Lifelong Options Program — registration and ‘out-reaching’ activities 

Innovative and developmental activities 

Organisation of learning (including differentiation, adaptation to needs of students, integrated teaching through interdisciplinary 
learning approaches etc.) 

Human resources policies (including continuing training of teachers) 

Innovation of education and development work 
Source: Van den Berghe (1998, pp.42-3) 

In his analysis of the various examples he reports, Van den Berghe (1998) shows how the earlier ‘message 
purpose model’ can account for most indicators. He suggests other dimensions on which indicators can 
be classified. However, the most useful is his alternative ‘layer model’, an example which is reproduced 
here as table 4. The details do not translate easily into the Australian context but the intention is apparent. 



Other performance measurement examples 

Blom and Meyers (2002) have provided a review of international perspectives on quality indicators in 
VET. They adopt a useful shift in terminology from performance indicators to quality indicators. This 
shift connects well to the broader approach to performance indicators taken in this report. Quality 
indicators are defined as performance indicators which focus on particular quality objectives. Quality 
objectives go beyond registration of outputs/ outcomes to include evaluation of inputs, expectations and 
experiences, both as mediating factors affecting outputs/ outcomes, and as features of quality delivery in 
their own right. 
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Table 4: The layer model for quality in VET — examples of areas for definition of quality indicators 

in VET provision 



Layers 


Quality of design 


Quality of conformance 




initial VET 


Continuing VET 


Initial VET 


Continuing VET 


Policy & 
implementation 


Adequacy of 
identification of 
training needs 

Effectiveness of 
translation of training 
needs into objectives 
and designs 


Adequacy of 
identification of 
training needs (but 
more likely to be 
micro-economic level) 


Cost-effective use of 
resources 

Adequate balance of 
training provision 


Systematic 
implementation of 
defined policy cost- 
effectiveness of 
training plan 


Institution 


Efficiency of the 
training plans in 
achieving desired 
results 

Suitability of premises 


Successful 
positioning in the 
market 

Adequate offer of 
training courses 


Efficiency of the 
training and financial 
management 

Staff motivation 


Cost-efficient delivery 

Quality assurance of 
processes 


Course program 


Relevance of content 
and methods 


Customer orientation 


Provision of training 
tools and materials 


Customer satisfaction 


Teacher/trainer 


Adequate skills and 
qualifications 


Relevant experience 


Performance 


Attitude towards 
trainees 


Student/trainee 


Adequate 
qualifications and 
preparation 


Motivation 


Trainees’ success 
rates 


Meeting needs of 
employment 



Source: Van den Berghe (1998, p.47) 



Some international approaches and activities identified by Blom and Meyers (2002) are pertinent to the 
concerns of this report. Of particular interest are developments in Europe, United Kingdom, New 
Zealand and Scotland. 

The Organisation for European Co-operation and Development (OECD) (2000) identifies a number of 
key factors for successful transition from education to work — only a part, of course, of training delivery. 
While two are concerned with matters beyond the control of training institutions (such as the economy 
and national policy formation), four factors are pertinent to our interests in institutional decision-making. 
These are: 

A pathways from education to work (which could be translated into institutional ‘work finding 
activities’) 

A opportunities to combine study and workplace experience (which could be translated into institutional 
‘work— study accommodation’) 

A safety nets for those at risk (which could be translated into ‘at-risk identification and support’) 

A effective information and guidance systems. 

In each case, the question can be asked ‘how well does the institution do this?’. This is a useful 
supplement to other aspects of training delivery already canvassed. 

A slightly different perspective on key factors in training delivery is that of Seyfried (1998) in relation to 
the United Kingdom: relevance of the training provided; confidence in the standards of training; 
credibility of the qualifications; competence of the teachers; and flexibility and cost-effectiveness. 
Although these are framed as system-level indicators, they can be reframed for institution-level 
consideration by the use of appropriate measures. 

The New Zealand Qualifications Authority has developed a system of quality management and self- 
review for provider institutions. For this, the focus is on ensuring that: 

A There is a planned and systematic approach to achieving the organisation’s goals and objectives. 

A Cultural and other expectations of clients (including learners) are taken into account. 
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4* Information is obtained, reported and presented in a manner appropriate to clients (including 
learners). 

4* The organisation is substantially achieving its goals and objectives. 

(New Zealand Qualifications Authority no date, p.7) 

The self-review is also characterised as an internal quality audit serving two purposes: demonstration of 
compliance with stated requirements; and effectiveness against indicators of good practice. It is stated 
that verification and sufficiency of evidence will be substantiated using a number of techniques: 

4* examination of documents and records 
4* interviews and observation 
4* sampling 

4* corroborative evidence from different sources. 

(New Zealand Qualifications Authority no date, p.7) 

There are three components of the required standard: 

4* The provider has measurable goals and objectives for education and training. 

4* The provider puts into practice quality management systems to achieve its goals and objectives. 

4* The provider is achieving its goals and objectives, and can assure that it will continue to do so. 

(New Zealand Qualifications Authority 2002, p.8) 

It is expected that evidence be collected about each of these. Specific guidelines for each are provided in 
the form of subcategories. While these are quite detailed, they are mainly concerned with satisfying 
regulatory requirements. For example, it is expected that provider institutions will develop and apply 
performance indicators for evaluating success in reaching goals and objectives. However, further guidance 
is not provided on developing these indicators. A useful evidence matrix is provided which shows what 
kinds of evidence would be appropriate for each of the components of the standard (New Zealand 
Qualifications Authority no date). 

The Scottish Quality Management System (2002) is another example of a total quality management 
system serving, as for the New Zealand system, both an accreditation function and an improvement 
function. However, again, it is directed at the managerial functions to accomplish this rather than the 
substantive issues of what performance indicators to use. The current system adopts ten ‘standards’ 
(replacing the previous 14): 

1. Strategic management The organisation has a clear sense of purpose and direction. 

2. Quality management. The quality system ensures that clients, learners and staff needs are met. 

3. Marketing and customer care-. The needs of the organisation’s clients and learners are 
identified, its education and training services are effectively promoted and the needs of 
clients and learners satisfied. 

4. Unman resources and development. The structure, level, and type of staffing is appropriate for 
the education and training services provided. Staff development provision meets the needs 
of both the organisation and the individual. 

5. Hqual opportunities-. Equal opportunities are ensured for all clients, learners, and staff. 

6. Health and safety. There is a safe and healthy environment for all learners, staff and visitors. 

7. Communication and administration-. Communication and administration arrangements meet the needs 
of the organisation, external bodies, clients, learners and staff. 

8. Guidance services. The needs of individual learners are identified, formulated, progress reviewed 
and support provided where needed. 

9. Program design and delivery. Program design is effective when the program’s content and outcomes 
are relevant and encourage access. Program delivery is effective when delivery methods are 
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appropriate and varied, emphasise activity and responsibility, and are responsive to the needs of 
the learner. 

10. Assessment for certification-. Assessment for certification confirms that the learner has achieved the 
standards required by the awarding body for the award. 

These other examples of performance indicators do not provide much further elaboration of the 
examples previously provided. The concentration on total quality management is of some interest in that 
there is typically an extensive set of guidelines for implementation of such systems. However, the 
emphasis is on the management functions rather than on the evaluative and improvement aspects of 
institutional health and success. Such systems address surface features rather than substantive issues, and 
are ad hoc and atheoretical in their identification of organisational features. There is benefit in stressing 
the more substantive issues of the earlier examples of performance measurement. The following section 
also asks the question: what aspects of organisational structures and operation appear to matter. 



Organisational performance measures 

The following discussion focuses on the research literature relating to organisational culture, climate, 
change and effectiveness. The aim was to search for ways in which institutional performance indices might 
be broadened into a more comprehensive set of indicators for evaluating individual institutions and 
monitoring institutional health. This could best be done by looking beyond the usual literature on 
performance indicators. 

It has already been noted that much of the literature on performance indicators deals with indicators in a 
general way and fails to reveal how such indicators could be operationalised as measures. The following 
discussion serves as the basis for the subsequent discussion of performance measures in the next chapter 
of this report. The intention is to push beyond generalities towards measures which could be used to 
collect data for monitoring and evaluating institutional health and success. 

The discussion in this section on organisational performance is organised under three headings: 
organisational culture and climate; organisational change; and organisational effectiveness. It is important 
to note that this is not a comprehensive analysis of these issues. Rather, the discussion focuses on 
interesting possibilities for institutional characteristics which can be observed and monitored and which 
are relevant to success and effectiveness in organisational functioning. These characteristics may be inputs, 
mediating characteristics or outputs/ outcomes, although here they are mainly mediating characteristics (or 
processes). A distinction can also be drawn between objective measures (things that can be counted) and 
subjective measures (requiring human observation or reaction, in some cases relating to personal 
experiences and feelings). 



Organisational culture and/ or climate 

Organisational culture is recognised as an important determinant of organisational performance. Schein 
(1985) provides a definition of organisational culture which reinforces the connection between culture 
and performance: 

Organizational culture is the pattern of basic assumptions that a given group has invented, 
discovered or developed in learning to cope with its problems of external adaptation and 
internal integration, and that have worked well enough to be considered valid, and therefore, to 
be taught to new members as the correct way to perceive, think and feel in relation to these 
problems. (Shein 1985, p.6) 

In other words, organisations function with a framework of assumptions, understandings and practices 
which have become implicit among those who work in the organisation. This framework may be 
construed as an organisation’s ‘culture’. Because such assumptions, understandings and practices are 
considered ‘natural’ and ‘necessary’, they are difficult to divest, even when circumstances call for rapid 
change. Consequently, mechanisms are needed to monitor the characteristics of an organisation’s culture. 



NCVER 



27 




Sparrow (2001) lists several measures or steps which may be undertaken to monitor the culture of an 
organisation and its effects. He suggests several steps that can be taken to improve the effectiveness of 
the organisational culture. These include: 

A Assess the extent to which any cultural change initiative has taken effect. 

A Assess the extent to which various control systems operate consistently. 

A Assess the need for team-building across different organisational units (sub-cultures). 

A Assess the consistency of assumptions about desired change across the organisation. 

A Assess the cultural imperatives that any change process must take into account. 

A Assess the management competencies (attitudes, mindsets) needed for future success. 

Litwin and Stringer (1968), in one of the seminal works on the topic, define organisational climate as ‘. . . a 
set of measurable properties of the work environment, based on the collective perceptions of the people 
who live and work in the environment and demonstrated to influence their behaviour’ (p.l). While 
conceptions of culture and climate are often difficult to discern in the literature (Hoy, Tarter & Kottkamp 
1991), a useful distinction can be made, in that culture relates to shared assumptions and beliefs, whereas 
climate relates more narrowly to shared perceptions of behaviour (Ashforth 1985). Consequendy, climate 
studies provide more direct linkage to the design of associated indicators. Because they relate to perceived 
behaviours, they are also more amenable to measurement (Lundberg 2001). In other words, studies of 
organisational culture tend to be more qualitative, while explorations of organisational climate tend to be 
more quantitative. Culture studies adopt a stronger phenomenological approach in contrast to the more 
measurable characteristics of climate studies. 

In the arena of school organisational climate studies, several measures have been isolated as significant 
indicators at the level of the individual institutions. Brookover et al. (1978) developed school climate 
studies as a means of accounting for various educational processes which had been ignored in earlier 
studies. His focus was upon social climate. 

In that study, student-level measures covered: 

A sense of academic purpose 

A future evaluations and expectations 

A perceived present evaluations and expectations 

A perception of teacher push and teacher norms 

A academic norms. (Brookover et al. 1978) 

Teacher-level measures covered: 

A ability, evaluations, expectations and quality of education/ college 
A present evaluations and expectations for high school completion 
A teacher— student commitment to improve 
A perceptions of principal’s expectations 

A academic purpose. (Brookover et al. 1978) 
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Principal-level measures covered: 

A parent concern and expectations for quality education 
A efforts to improve 

A principal and parent evaluation of present school quality 

A present evaluations and expectations of students. (Brookover et al. 1978) 

The Brookover et al. (1978) study focused on elementary schools and found that the variable which had 
the largest correlation with student achievement was student sense of academic purpose (actually 
represented negatively as sense of academic futility) or student feelings that they were able to succeed 
within the institution. Other indicators found to be significant in the Brookover et al. (1978) study were 
students’ perceived present evaluations and expectations by their teachers, and teachers’ present 
evaluations and expectations for high school completion. This study indicates the importance of the 
perceptions of individual students, as well as the instructor’s evaluations of students and their 
expectations about whether students will complete their courses. Such perceptions are a key determinant 
of success for the individual student, and consequently for institutional performance. 

In a later study, Hoy, Tarter and Kottkamp (1991) isolated five dimensions of secondary school climate 
which related specifically to the behaviour of the principal administrator and the teachers. These 
dimensions have a degree of generality and may be seen to be relevant across the VET sector. Principal 
behaviour was divided into ‘directive principal behaviour’ and ‘supportive principal behaviour’. 
Dimensions relating to teacher behaviour were found to cluster into three groups: ‘engaged teacher 
behaviour’, ‘frustrated teacher behaviour’ and ‘intimate teacher behaviour’. 

Supportive principal behaviour sample items included that the principal set an example by working hard, 
used constructive criticism when addressing staff members and explained why criticisms were directed 
towards teachers (Hoy, Tarter & Kottkamp 1991, p.55). Directive principal behaviours included the 
principal exercising strong control, supervising teachers and closely monitoring all aspects of their work 
(p.55). Such measures may be generalisable to the VET sector and applicable to institutional directors/ 
chief executive officers / principals. 

This is also the case for teachers/instructors in individual VET provider institutions. Items associated 
with engaged teacher behaviours included teachers supporting/ assisting one another; teacher friendliness 
towards students; and allocating time to students with specific needs/ problems (Hoy, Tarter & Kottkamp 
1991, p.55). Frustrated teacher behaviour items related to the degree to which teachers’ mannerisms were 
annoying; paperwork was overwhelming to teachers; and teachers’ non-teaching duties considered 
excessive (p.55). The final dimension, intimate teacher behaviours, related to such sample items as 
teachers’ closest friends being members of their institution’s staff; teachers inviting other faculty members 
to their home; and teachers meeting frequently to socialise with one another (p.55). Such measures 
provide a general set of factors, many of which may provide some useful insights into the nature of 
personnel relations within individual VET institutions. 

In the VET sector, a recent study of high-performing individuals and organisations has identified 
important cultural features at the level of individual organisations (Mitchell & Wood 2001). High- 
performing VET institutions were those which developed collaborations with industry, were strategic 
about human resource development and were innovative in their approach to the challenges arising from 
the implementation of the National Training Framework (Mitchell & Wood 2001). The characteristics of 
high-performing VET organisations are listed in table 5. 
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Table 5: Summary of characteristics of high-performing VET organisations, identified in case studies of 

Framing the Future projects, 1999-2000 



General characteristics Specific characteristics 



Strategic approach to human resource 
development 



Innovative response to challenges 



Collaborative arrangements with 
industry 



Staff development is valued at a strategic level. 

Staff development activities are provided in a range of key areas, each of which 
relate to specific organisational goals. 

A staff development unit, or special research and development unit, or other 
specialist group, oversees and coordinates all staff development activities across 
the organisation. 

Staff development is targeted at and designed for specific staff, e.g. sessional staff. 

Staff development activities involve a variety of participants from different levels of 
the organisation. 

A challenge or opportunity is approached with creativity and courage. 

Flexible delivery is used to overcome the problems of distance and insufficient 
funding. 

Staff are provided with opportunities to learn and develop through staff 
development activities designed to lead to further innovation. 

Training delivery is promoted and marketed to industry. 

Links with local companies are established to provide training and assessment 
services. 

Real business activity is stimulated within institute-based training programs, using 
‘practice firms’ sponsored by local industry. 

Flexible, innovative, quality training is provided to industries to ensure continued 
success. 

Contracts for the provision of training with national companies are pursued. 



Source: Mitchell and Wood (2001 , table 3) 



Organisational change 

In addition to the work on culture and climate, a considerable body of literature is devoted to 
organisational change. Given the increasingly volatile and dynamic environment in which both public and 
private organisations must operate, the capacity to be able to alter practice and to change is imperative. 
This, however, is no easy matter. The critical message from two of the world’s leading theorists (and 
practitioners) on organisational change and reform is that ‘. . . there is no one right approach to change, 
effective in all circumstances’ (Stace & Dunphy 2001, p.4). They indicate that this also applies to the rate 
of the change process and provide a list of dilemmas which currently challenge organisations (p.6): 

A Should change be adaptive or involve rational strategy development? 

A Is cultural or structural change required? 

A Is there a need for continuous improvement or radical transformation? 

A Should principles of empowerment be adopted or those of leadership and command? 

“y* Are we interested in economic or social goals? 

Such a list is presented in terms of binaries — yes or no — rather than more complex options. Limerick, 
Cunnington and Crowther (1998) challenge such questions by interpreting the world from a postmodern 
perspective, referring to a postcorporate world in which the ‘new, postcorporate, network organisation’: 

A has evolved to deal with a new era of change 

A reflects broader patterns of social change 

A has a radically different pattern of organisation 

A has a subtly different corporate culture 

A requires a new, strategic mindset 

A is participant focused, not manager focused. (Limerick, Cunnington & Crowther 1998, pp.3—4) 
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This highlights the complexity and difficulties associated with organisational reform. To be efficacious, 
such reform must, in the first instance, be cognisant of the current contextual circumstances of the 
organisation. Denison (2001) identifies three factors which need to be considered when initiating change. 
He argues for ‘taking the native’s point of view seriously’ (p.348). By this, he means that it is necessary to 
present change initiatives in a sufficiently instrumental way to elicit understanding and then to provide 
support for change. Secondly, he advocates a ‘system perspective’. Change is dependent upon being able 
to successfully ‘map backwards’ from the visible manifestations within an organisation to the values which 
are publicly expressed by those in the organisation, and ultimately to the assumptions underlying those 
values (p.351). Leverage for change may only be maximised if there is a clear articulation between the 
outcomes of an organisation, these expressed values and their underlying assumptions. Anomalies may 
become apparent if these links are not evident, or if one or more elements of the system is not readily 
discernible. 

While acknowledging the uniqueness of any given unit, Denison (2001) also advocates the importance of 
benchmarking within institutions. He states that, while it is difficult to compare according to the 
underlying assumptions within any given unit of an organisation, it should be possible to compare at the 
level of espoused values. The emphasis upon espoused values also highlights the role of personnel within 
organisations. 

Waldman and Atwater (1998) emphasise the role of personnel in the change process via their concept of 
‘360 degree feedback’. The concept of ‘360 degrees’ refers to the individual’s capacity to draw feedback 
about performance from everyone around them; that is, being able to draw upon perceptions of those 
‘above,’ ‘below’ and ‘beside’. However, Waldman and Atwater (1998) are aware that this is not a universal 
panacea. Potential concerns revolve around fears of retribution from managers receiving poor ratings, 
defensiveness and denial from individuals receiving low ratings, conflicting ratings, the potential for self- 
esteem problems, issues of validity associated with ‘game playing’ (not taking process seriously or rigging 
the process), concerns about the time and cost involved in procuring responses, increased expectations 
coupled with a lack of change, ‘faddism’ and concerns about follow-up procedures (pp.12— 17). 
Nevertheless, it is potentially a powerful means of collecting information about individual performance 
and would appear to be of most value if utilised as a means of collecting data for professional 
development purposes rather than for appraisal purposes. 

The literature concerning change as it relates to schools is also informative. The school reform literature is 
perhaps the best source of material about change in educational institutions. One of the latest studies on 
school reform, which highlights the complexity of the change process, is the Queensland School Reform 
Longitudinal Study (QSRLS). The findings (2001) include institutional characteristics which lead to 
improved student outcomes. One of the important levers for change which may be extrapolated to the 
VET sector from the Queensland study is organisational capacity. The concept of organisational capacity 
is underpinned by the notion that learning is influenced by the way education institutions are organised. 

The 2001 Queensland study found that students’ academic and social outcomes improved in those 
schools in which teaching and assessment practices were more productive, the curriculum was more 
connected with student background and real world experiences, classroom environments were more 
supportive and there was a greater recognition of difference between individual students. Specific 
elements associated with teachers’ collective responsibility for student learning, the degree of professional 
community within a school and the extent to which leadership in the school was focused on issues of 
teaching and learning were associated with more effective and appropriate teaching and assessment. 
Collectively, these features are referred to as organisational capacity and signal ramifications for the VET 
sector, because each of these elements is readily transferable to a range of VET institutional settings. 



Organisational capacity 

The concept of organisational capacity is expanded by King, Ladwig and Lingard (2001). This work is 
based upon earlier work undertaken at the University of Wisconsin, as part of the Centre on the 
Organisation and Restructuring of Schools project (Newmann and Associates 1996). They point out that 
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there is a high degree of complexity inherent within the notion of school organisational capacity and that 
failure to recognise this complexity has led to a sustained history of failed reforms in the schooling sector: 

School reform and restructuring movements have long focused on changing 
organisational features of schools as a means to school improvement. Research into the 
organisational efficacy of such initiatives has raised serious questions about the manner 
and form of common school level organisational changes . . . often find that typical 
changes lead to little or no real improvement in student learning outcomes. 

Developments in school reform research have drawn from such prior research to 
suggest that a more extensive and complex understanding of schools as organisations is 
necessary if reform initiatives are to be more efficacious. 

(King, Ladwig & Lingard 2001, p. 1 1 6) 

King, Ladwig and Lingard (2001) summarise the organisational features in schools which have the 
potential to lead to improved student outcomes. These include principal leadership, formal structures of 
organisations, site-based management, teacher collaboration, and improving the dynamics between 
teachers and students. They highlight a model of school organisational capacity which consists of the 
following features: professional capacities of teachers; professional learning community within schools; 
program coherence; leadership; and technical resources (pp.125— 8). Each of these elements will be 
considered in turn and their pertinence to the VET sector will be explored. 

Professional capacities of teachers 

The knowledge, skills and dispositions of individual teachers/instructors is most important. The notion 
of teacher capital (an extension of Bourdieu’s notion of capitals [1986, 1991, 2000]) is an important 
concept within the Queensland School Reform Longitudinal Study findings. It focuses upon the extent to 
which teachers believed that they were responsible for student learning, their conception of their role, the 
degree of autonomy they believed they exercised and their view of knowledge (Queensland School 
Reform Longitudinal Study 2001, pp.96—101). These indicators showed that those teachers who rated 
highest in classroom practices also felt more responsible for their students’ learning, believed that their 
role was predominantly that of facilitator of knowledge (rather than an explainer or ‘director’ of class 
activities) and focused more strongly on skills and concepts rather than content. Teachers who scored 
lowest in classroom practices felt little responsibility for their students’ learning and considered that their 
role was as an explainer and sometimes a director of class activities rather than as a facilitator. 
Furthermore, they believed they exercised relatively little autonomy within their school and they focused 
much more strongly on content than on skills or concepts. This has implications for VET sector staff 
development, which is currently seen as an area requiring a much more substantial emphasis (Mitchell & 
Young 2001). 

These results are also reflected in the more prescriptive literature on individual teachers/instructors. 
Darling-Hammond (1998) outlines that teachers require knowledge relating to: 

A subject matter (which should be understood deeply to enhance students’ abilities to develop effective 
cognitive maps) 

A pedagogical content knowledge, with a focus on how to connect knowledge across fields for students 
and to improve accessibility 

A what students currently know and are capable of doing and how they best understand key concepts 

A child and adolescent development in the areas of the cognitive, social, physical and emotional 

A social background relating to culture, family experiences, developed intelligences and approaches to 
learning 

p- knowledge about learning, in that different teaching strategies are necessary to engage students with 
particular types of material for different purposes and in different contexts 
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A curriculum resources and technologies to enable students to interact with sources of information, to 
engage in higher-order thinking processes, including collaborative practices which seek to encourage 
students to do so. 

Darling-Hammond (1998) also noted the importance of teachers understanding how to analyse and 
reflect upon their practice and to determine the effectiveness of their work as a means of further refining 
their practices. 

These thresholds of knowledge and skills may be enhanced by adopting targeted professional 
development of teachers. Consequently, in terms of improving individual institutions, performance 
indicators which focus upon knowledge and skills such as those may be instructive. The extent to which 
professional development programs exist which provide the capacity for individual teachers to isolate 
their own needs and to redress them may also constitute an effective institutional performance indicator. 

Professional learning community 

Another set of performance indicators may be developed around the notion of ‘professional community’ 
in individual institutions. This concept was originally identified by Louis, Kruse and Marks (1996) who 
highlighted the importance of the individual school as the vital unit for reform. For them, a ‘teacher 
professional community’ was one which involved a whole-of-school focus, as opposed to a sense of 
collegiality around other constructs within the school environment, such as particular knowledge/ subject 
domains. As they put it: 

An entire faculty comes together around meaningful, shared issues irrespective of teachers’ 
individual disciplines. Schoolwide community does not devalue other forms of collective 
professional relationships nor is it incompatible with a departmental structure. But it does entail 
staff members’ taking collective responsibility for achieving a shared educational purpose for the 
school as a whole and collaborating with one another to attain it. 

(Louis, Kruse & Marks 1996, p.180) 

Louis, Kruse and Marks (1996) argue that, even though individual teachers have a significant impact upon 
students, it is the collaborative effort of teachers, focusing on the promotion of high levels of student 
learning, which ultimately has the most benefit for students. Instructors working in isolation from one 
another are less likely to be able to effect and sustain the sort of substantive change necessary to 
maximise student learning. This sentiment may seem self-evident, but much of what occurs within 
schools seems contrary. This is particularly the case in secondary schools (Little 1990). This finding may 
be extrapolated across educational sectors, even though specific contextual factors may be quite different. 

Under the rubric of teacher professional community, Louis, Kruse and Marks (1996) have isolated a 
number of factors which they believe are instrumental in developing such communities. The critical 
factors which characterise these entities include a focus on shared norms and values between members of 
staff, a focus on student learning, collaboration, deprivatised practice and reflective dialogue. 

Without shared norms and values, individuals within an organisation are at liberty to do as they please 
(within the normal professional and other ethico-legal parameters). This does not necessarily have 
beneficial outcomes for students. In contrast, when shared values do exist among teachers, expectations 
may be higher and teachers may feel more empowered to address difficult issues. This helps to mitigate 
any deleterious external factors which may impinge upon the life chances of students (such as socio- 
economic status, ethnicity). Teachers experience a sense of shared efficacy, which can reinforce the belief 
that students can achieve, regardless of circumstances (Louis, Marks & Kruse 1996). 

Reflective dialogue also heightens teachers’ awareness of their practice and enables them to discuss 
practice openly with colleagues. Such dialogue may cover content issues, teaching strategies, assessment 
practices and background factors affecting student performance (including socio-economic and ethnic 
factors) (Louis, Kruse & Marks 1996). 

This is also related to the concept of deprivatisation of practice. This refers to teachers using one another 
for insights into how best to assist students (Louis, Kruse & Marks 1996). It also requires a degree of 
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trust, such that teachers are prepared to practise their craft in a more public arena than is traditionally the 
case. 

A natural extension of reflective dialogue in conjunction with deprivatised practice is increased 
collaboration between teachers. Teachers are much more likely to be successful with students if they can 
draw upon the expertise and insights of colleagues. Such collaboration may involve planning the 
curriculum in conjunction with others, co-teaching, observing one another and moderating one another’s 
assessment of student work (Louis, Kruse & Marks 1996). 

These findings on the importance of professional community in restructuring schools indicate that wide 
variations exist in the extent to which professional communities are evident in schools. Furthermore, the 
elements of professional community are developed to varying extents because of particular structural 
features in schools, as well as personnel issues. Structural factors relate to school size, staffing complexity, 
the availability of scheduled planning time and teacher empowerment, all of which are critical for 
promoting effective professional community. Similarly, human and social resources, specifically in relation 
to whether or not there is a supportive leadership, openness to innovation, respect, feedback on 
instructional performance and approach to professional development, affect levels of community. Louis, 
Marks and Kruse (1996) point out that, while professional community is an observable entity, there are 
some situations in which it becomes more apparent than others. Many of the elements which underpin it 
are ‘manipulable’, in the sense that policy and administrative practices can promote (or inhibit) them. 

Program coherence 

Programs for student and staff learning must be aligned. Schools experiencing difficulties in improving 
learning opportunities for students and staff are often characterised by the unfocused pursuit of multiple 
objectives, which then crystalise into a multiplicity of programs. In this situation, new ideas are pursued 
rather desperately in a relatively haphazard and ultimately unsustained manner (King, Ladwig & Lingard 
2001). The opposite of this is a school program in which student and staff learning are aligned, according 
to the same whole-school goals and implemented in a long-term manner. 

Leadership 

King, Ladwig and Lingard (2001) emphasise the importance of principal leadership within schools. This is 
because legal responsibility resides primarily with the principal and because research into the role of the 
principal indicates the critical role this person plays in the life of the school. However, a broader concept 
is educational leadership. 

Educational leadership is characterised by complexity. It requires more than merely the leadership 
provided by the principal or director of an educational organisation. Educational leadership involves 
teachers not simply administrators (Christie & Lingard 2001). Crowther et al. (2002) also reinforce the 
need to consider leadership beyond the purely administrative role of the principal or director. These 
authors point to teacher leadership as a means of reforming schools and improving the quality of 
teaching (see also Newmann and Associates 1996). However, as the Queensland School Reform 
Longitudinal Study found, for improved student performance, it is necessary to ensure that all elements 
of school organisational capacity are present (Queensland School Reform Longitudinal Study 2001, p.38) 
not just leadership functions. 

Technical resources 

High-quality curricula, up-to-date and substantive texts and other paper and digital sources of 
information, instructional resources, assessment items and physical plant are necessary to achieve 
exemplary standards. Such factors are a necessary starting point to achieving an effective holistic 
educational environment. Institutions with exemplary organisational capacity should be able to provide 
evidence that renewal of technical resources is undertaken in a sustained and coherent manner. 
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Organisational effectiveness 

Research on organisational effectiveness and school effectiveness is relevant to the success and health of 
VET institutions. Van den Berghe (1998), dealing with quality indicators in the vocational sector, draws on 
the work of Mortimore et al. (1988) who report the findings of research on school effectiveness. 

Both Van den Berghe (1998) and Mortimore et al. (1998) point to the inadequacies of the traditional 
input/ output paradigm which ignores mediating processes. Reynolds and Teddlie (2000) also refer to the 
problems associated with this paradigm. Inputs relating to resource variables (such as per pupil 
expenditure) and student background characteristics (particularly in relation to socio-economic status) 
were traditionally used to predict outputs. Consequendy, the findings of early major economic and 
sociological studies dealt with a very small number of indicators in what is now recognised as a simplistic 
manner, and did not relate to a sufficiently broad array of variables or the inter-relationship between 
variables to be sufficiently useful. 

Consequently, more sensitive measures of classroom input are beginning to be used, specifically, those 
which relate to input from the classroom (teacher) level, rather than simply from the whole-school level 
(Reynolds & Teddlie 2000). Student-level input variables enable finer-grained analyses to be undertaken 
and provide a more accurate level of correlation between student -level input measures and student-level 
output measures. 

The research of Summers and Wolfe (1977) and Murnane (1975) is important in highlighting the inputs 
of various teachers and the impact they have upon particular students. Although such studies do not 
necessarily report the accumulated effect of whole-school/institution factors, one identified determinant 
is the quality of the college attended by the teacher (Summers & Wolfe 1977). In other words, preparatory 
education and training is important. 

A later phase of the effective schools movement shifted the focus from merely describing characteristic 
features and potential performance indicators to creating effective schools, particularly for the urban poor 
(Reynolds & Teddlie 2000). The five factors identified in this new ‘school improvement movement’ were: 

A strong instructional leadership from the principal 
A a pervasive and broadly understood instructional focus 
A a safe and orderly school learning environment or ‘climate’ 

A high expectations for achievement from all students 

A the use of student achievement test data for evaluating program and school success. 

(Reynolds & Teddlie 2000, p.10) 

The downside of this phase of studies (and earlier work in the school effectiveness research) was that 
sampling and analysis strategies were skewed in pursuit of the equity agenda. Consequently, institutional 
contexts were not considered. More recently, there has been an increased methodological complexity 
which takes account of contextual factors. Differences in school effects across different school contexts 
were identified. New studies focused upon factors which caused differences across different contexts, but 
still maintained notions of inclusiveness and equity because there was an emphasis upon improvement 
across all contexts (Reynolds & Teddlie 2000). Regardless of the nature of the school, the key question 
became: ‘How can we produce better schools for any and all students?’ 

While the school effectiveness literature provides a number of appropriate indicators of performance, it 
also cautions against relying too heavily upon specific lists, or assumptions that it is possible to develop 
finite lists of performance indicators. It is important also to realise that the search for appropriate 
indicators is an ongoing and often elusive exercise. Wilcox (1990) refers to distilling them as akin to ‘... the 
search for the Holy Grail . . .’ (p.3 1). He argues that there is a need for a limited set of quality criteria, 
rather than a vast array of indicators which are of limited value beyond specific educational contexts 
(within which they may well be useful). He provides an example from the United Kingdom Department 
of Education and Science which recommended six key indicators: 

A staff— student ratio 
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A non-teacher costs per full-time equivalent student 
“y* cost per full-time student 
A course completion rates 

A rates of qualifications gained and cost per qualified full-time student 

A rates of employment or progression to further or higher education for completing students. 

(Wilcox 1990, p.33) 

The first three of these can be considered as inputs (to institutional delivery) and the last three as outputs. 

Given the strong educational focus of VET organisations, it is important to consider the specific role of 
teaching as an indicator of performance. The quality of training and learning has been reported to be 
satisfactory in many VET sites, but there are currently many examples of instances in which this is not 
the case (Schofield 2000). In a recent review of the quality of training in Victoria’s apprenticeship and 
traineeship scheme, it was noted that 20% of trainees did not believe that they were learning new skills 
(Schofield 2000). 

This situation is not limited to the VET sector. Within the university sector, client dissatisfaction, 
combined with broader global economic imperatives have also resulted in a heavier emphasis upon the 
role of teaching. Consequently, universities have been challenged to redress concerns about teaching 
quality and student learning. To this end, many have developed specific sets of priorities and strategies 
which can be subsequently reformulated into individual institutional indicators of teaching quality and 
learning performance. For example, the University of Queensland has developed a number of such 
indicators to cover students, staff, programs and the learning environment, as shown in the table 6. 



Table 6: Priorities and indicators adapted from the University of Queensland 

Priority 1 : Ensure that students are adequately prepared and supported at transition phases 

Indicators: 

Provision and nature of induction activities, including high-quality advice on course and subject selection and activities that 
facilitate social interaction 

Provision of opportunities to facilitate students' independent critical learning 

Prepare students for relevant and appropriate use of information and information technology 

Provision of systems to glean and respond to feedback from undergraduate and postgraduate students in terms of their 
experiences during their transition phases. 

Priority 2: Enhance the quality of the continuing learning experience 

Indicators: 

Provision to undertake regular student feedback of institution course experiences through questionnaires, student focus groups 
Capacity to continue to reinforce the relevant and appropriate use of information and information technology 
Provision to ensure students receive fair, equitable and appropriate assessment 

Provision of systems to enable all students to have access to diagnostic and corrective feedback on assessment, as part of the 
learning process 

Provision of course and subject selection processes and study support for continuing students through high-quality course 
advising, mentoring, supervision and other appropriate activities 
Provision of strategies to identify students at risk of exiting courses early 
Provision of opportunities to enhance the quality of workplace experiences. 

Priority 3: To ensure that students’ experiences are nationally and internationally relevant 

Indicator: 

Provision of systems to monitor the quality of student experiences in terms of national and international best practice. 

Priority 4: Continue to ensure that the institution attracts and retains staff with excellent teaching skills 

Indicators: 

Appropriateness of selection criteria for all instructors 

Extent to which teaching ability is utilised for confirmation of continuing employment and career advancement. 

Priority 5: Enhance teaching skills through initial and continuing professional development of academic staff 

Indicators: 

Provision of initial and ongoing professional development courses 

Extent of a culture of peer support and mentoring for improved professional practice 

Provision of systems to maintain instructor's technology skills for teaching purposes. 

Priority 6: Continue to support, encourage and reward teaching excellence 

Indicator: 

Provision of a system to encourage innovation in teaching practice that may include technological innovation within curriculum 
design. 
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Priority 7: Implement an institution-wide system for improving course and subject quality 

Indicators: 

Monitoring and reviewing the quality of courses against courses offered nationally and internationally, including feedback from 
professional bodies, public and private sector employers 

Implement regular monitoring by departments and schools of assessment requirements and practices in all subjects and 
regular reviews of subject profiles. 

Priority 8: Continue to recognise the importance of the learning environment 

Indicators: 

Support the ongoing improvement of teaching facilities for adaptable use 
Promote flexible modes of teaching based on best practice across all campuses 

Continue to develop increased opportunities for students to study using flexible teaching and administrative methods. 

Source: Adapted from University of Queensland (2000) 

This focus on building better educational institutions for all students resonates with current efforts within 
the VET sector, and particularly within TAFE institutions. Some evidence of how this might be achieved 
is provided in a recent study which considers the impact of the Framing the Future staff development 
initiative, initially designed to implement the ANTA National Training Framework (Mitchell & Wood 
2001). It draws upon the insights of high-skilled VET practitioners (and high-performing VET 
organisations) to determine how VET organisations have instigated long-term change. The study found 
that highly skilled VET practitioners had particular characteristics and skills relating to strategic 
management, change management, staff development, implementation of training packages and 
networking. A summary of these skills is presented in table 7 (from Mitchell & Wood 2001). 

Table 7: High-level skills of VET practitioners 

Skill area Specific skills 

Strategic management Analyse external environment and identify challenges and opportunities. 

Analyse the internal skills and resources and identify training and development needs 
required to meet the challenges of the external environment. 

Develop objectives to meet the challenges of the external environment. 

Choose and implement strategies for training and development in order to achieve 
organisational objectives. 

Change management Create readiness for change within the VET organisation by encouraging professional 

development activities. 

Overcome resistance to change through addressing various impediments to change. 
Manage organisational transition from its present state to the desired state. 

Provide expert facilitation to guide debate and discussion of change. 

Identify staff development needs of the organisation. 

Design staff development activities to meet individual and organisational needs. 

Promote continuous learning at the individual, team and organisational level. 

Encourage networking amongst staff as a means of sharing knowledge and skills. 

Provide leadership and guidance for staff undertaking staff development activities. 

Act as agents of change, promoting new skills and knowledge. 

Analyse and unpack’ training packages in order to increase knowledge and understanding. 
Develop strategies for the implementation of training packages. 

Establish partnerships with industry, other registered training organisations, New 
Apprenticeship Centres, to facilitate implementation of training packages. 

Identify the organisational value of networking. 

Establish contact with other registered training organisations across Australia in order to 
share knowledge, ideas and resources. 

Manage and sustain networks through email contact, forums and workshops. 

Source: Mitchell & Wood (2001 , table 2, pp.7-8) 



Design and implementation of 
effective staff development 



Creative interpretation and 
implementation of training 
packages 



Development and networks 
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Conclusion 



Currently, systemic data are collected across the VET system but not for the purpose of institutional 
monitoring and evaluation. A coherent set of key performance criteria applicable to individual institutions 
to assist them in improving current practice needs to be developed. Various pointers have been identified 
in the international literature on performance measurement but are incomplete. A model which reflects 
the significance of organisational culture/ climate, change, effectiveness and program delivery is proposed. 
The model has the potential to provide a framework for institutional self-monitoring and improvement. 
There are likely to be considerable benefits for the health of the whole VET system in moving in this 
direction. 

Institutional performance indicators are a necessary step in the development of a more comprehensive 
model to improve VET performance. It would seem that they also are a necessary part of the next stage 
of development of the VET system — what Veenker and Cummins (no date) recendy have called the 
needed ‘third revolution’ in vocational education and training. 
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Towards a model for monitoring and 
evaluating institutional performance 



This section describes a model which has the potential for monitoring and evaluating institutional 
performance. This model offers a framework of indices derived from the review in the previous section 
of relevant theories, concepts and practices in institutional monitoring and evaluation. It is not likely or 
possible (or intended) that all indices would be used on a regular basis. Nor are all indices of equal 
importance; rather, the listing offers a template from which indices might be selected to suit particular 
purposes and particular contexts. 

A distinction of purposes can be made between local and central (whether state or national). That is, a 
distinction can be made between the use of institutional indices for institutional self-monitoring and for 
central accountability. However, whether a distinction is necessary between state and national purposes is 
unclear. Some indices may be relevant both locally and centrally, and in this context it is possible to 
envision a hierarchy where all indices might be useable at institution level with only some of the indices 
being collated at state and national levels. 

Two points need to be made about the nature of these data. First, what is envisaged here are institutional 
profile data not individual student data. This might include some representation of sub-components of 
the institution (such as a field of study or department), but the indices themselves are at a level of 
aggregation beyond individual characteristics and responses. Some degree of data aggregation is essential 
to making sense of the characteristics and performance of the institution itself. It is contended here that 
aggregation across individual student data is best accomplished at the institution level, where 
interpretation of local idiosyncracies in the data can be identified and managed. 

Second, indices reported at state and/ or national level require explicit protocols to ensure that they have 
comparable meaning across institutions. Indices not reported at state or national level can be more 
specific to the institution. Choices can also be made among alternative indices according to local 
considerations, including, for example, any cycle of review adopted by the institution. If an institution 
engages in a comparative benchmarking exercise with other institutions, it would be necessary to establish 
a common approach to the collection of data on the chosen indices among those institutions. 

The indices in this framework can be categorised as either objective measures (quantitative data involving 
counting; for example, numbers of students or amount of money) or subjective measures (involving 
human judgment or report). Subjective measures can be categorical (such as responses to yes/no 
questions or checklists) or ratings (such as perceptions of the extent or intensity of some characteristic). 
Subjective responses can be collected as discursive comments or discussions, but these subsequently 
involve considerable effort and expense to analyse and code. It is best to use such data in exploratory and 
developmental phases of institutional evaluation, leading to the construction of defined checklists and 
rating scales. Databases are more efficient and useful if the data are coded numerically. 

Performance indices can also be categorised as referring to inputs, processes or outputs/ outcomes. Input 
indices cover characteristics of the institution which exist as a precondition for delivery of programs, such 
as staffing and facilities. Included here are financial indices, not in terms of available income, but rather in 
terms of its allocation and use within the institution. Process indices cover operational characteristics of 
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the institution which relate to the way the institution functions for the delivery of its training programs. 
Included in this category are organisational arrangements and procedures; for example, delivery 
mechanisms and change mechanisms, as well as less tangible but experiential aspects of the institution, 
such as its organisational culture and personal responsiveness. Output/ outcome indices cover those 
matters which can be considered to have resulted from the inputs and processes, whether deliberate or 
accidental, and whether intended or unintended. These may be immediate or longer-term; for example, 
module completion rates (immediate) or employer satisfaction (longer-term). 

In some cases, decisions need to be made about whether particular indices are inputs, processes or 
outputs/ outcomes. For example, student perceptions of the quality or relevance of their course might be 
considered as data relevant to inputs (for example, indicative of quality of the program staff, materials or 
facilities) or processes (for example, indicative of institutional adaptability to student needs, interests and 
concerns) or outputs/ outcomes (for example, indicative of their opinions and attitudes towards the 
institution — opinions and attitudes which may be expressed and disseminated through social contacts to 
the benefit or detriment of the institution and thus of importance for institutional image and survival). 
Such indices can be categorised in a multiplicity of ways. Each category draws a different implication from 
the data. The relative force of those implications depends on the context. 

The complete three-way categorisation (inputs/processes/outputs— outcomes x objective/subjective x 
local/ central) is shown in figure 1. What is shown in this figure is a three-dimensional structure in which 
the large box is made up of 12 smaller boxes. Each of the smaller boxes represents an inersection of 
three components, one from each side of the large box. For example, the small box at the lower left 
represents inputs— local— objective. Some cells in this figure may be blank, particularly for processes, which 
are likely to be restricted to local-use subjective measures. 

Figure 1 : Model of different use, characteristics and types of institutional indices 
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A full listing of the 12 categories of data represented in figure 1 would be: 



Use 


Objective/ subjective 


Input/processes/outcomes 


1 Local 


Objective 


Inputs/ Outcomes 


2 Local 


Objective 


Processes* 


3 Local 


Objective 


Outputs/ Outcomes 


4 Local 


Subjective 


Inputs 


5 Local 


Subjective 


Processes 


6 Local 


Subjective 


Outputs/ Outcomes 


7 Central 


Objective 


Inputs 


8 Central 


Objective 


Processes* 


9 Central 


Objective 


Outputs/ Outcomes 


10 Central 


Subjective 


Inputs 


11 Central 


Subjective 


Processes # 


12 Central 


Subjective 


Outputs/ Outcomes 



*probably non-existent; # probably not relevant 

Figure 2 exemplifies the relationships among various types of institutional indices. Thus, the institution is 
embedded in a socio-politico-economic context which affects the circumstances under which it 
operates — including the regulatory environment, the job market and the needs and aspirations of various 
stakeholders. The context has a direct influence on inputs, especially through government resources and 
controls, but also through student interest in and valuing of enrolment in training programs. 

Figure 2: A systems model of relationships among different factors in performance measurement 



Context 



Inputs 

T 

Mediating or process 
measures 

T 

Outputs/outcomes 
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The arrows in figure 2 represent causal links. Inputs can be resources, but they can also be the 
institutional plans and programs which the inputs make possible. Various features of the institution act 
(especially characteristics of the training programs) as mediating or process factors to provide the basis on 
which the institution brings various outputs and outcomes into effect. These factors are critical to the 
success of the institution; that is, to the quality of its operation. Such factors have too often been ignored 
in performance measurement, creating a ‘black box’ of missing information connecting inputs and 
outputs/ outcomes and making it difficult, if not impossible, to intervene with any measure of rationality 
to seek improvement in the outputs/ outcomes (except through manipulation of the inputs). 

A further issue concerns the source of subjective data. Among the various stakeholder groups, three main 
groups can be identified: staff, students and employers. Staff can be further classified as administration, 
teaching and support. Teachers, students and employers can be further classified according to field of 
study. Students can be identified by whether they belong to the various equity groups or not. Data should 
be tagged according to all of these categories. Whether analysis across all categories is appropriate will 
depend on the nature of the measure and the number of stakeholder cases. For example, some measures 
may concern the institution as a whole rather than the field of study. Furthermore, conclusions may be 
unreliable if based on few cases. 

While the primary concern here is with indices that can be collected systematically and continuously from 
the primary participants and interested parties (staff, students, employers), it is also possible to engage 
independent observers or evaluators, perhaps on a more periodic basis, to offer alternative viewpoints and 
analyses. These might be so-called ‘critical friends’ who identify ongoing issues which may need attention 
and assist in their resolution. Alternatively, they might be more ‘arms length’ from the institution, offering 
a completely independent and uncompromised viewpoint, perhaps through a formal report. Individuals 
can of course be replaced by teams. To some extent, this distinction is between ‘internal’ and ‘external’ 
evaluators. 

The following categorisation of institutional indices derives from the review of relevant background 
research given earlier in this report. This is incomplete in three ways. First, it has not been tested against 
the practical wisdom of VET administrators and teachers. Second, the list may need refinement to present 
a more manageable core of relevant indices. Third, the list requires three further components: definitions 
of each of the measures; exemplifications of the measures (for example, explicit rating scales); and 
protocols for their collection. In addition, further discussion is needed about how these indices might best 
be used for institutional monitoring and evaluation. 

Within the three meta-categories of input, process and output/ outcome measures, conceptually coherent 
sub-categories have been identified. For each sub-category, the likely origin of data for these measures is 
indicated. In some cases, mainly for input resources and outcome achievements, the data involve counting 
(amount of money and numbers of students). In other cases, particularly for process measures, the data 
may be based on human judgement. This might involve surveying particular stakeholder groups. 
Alternatively, it might involve collection of primary information, such as staff qualifications, which then 
needs to be collated by someone who makes an overall rating judgement. The person selected to 
undertake this role — management, staff or independent evaluator — would depend on the context and the 
development of protocols for the collection and interpretation of the data. 

Some rating scales, such as those involving qualitative ratings (of strength, extent or value) might assume 
(weak) equal -interval scaling (that is, an equal quantum of meaning between scale points). However, this is 
neither essential nor desirable in all cases. In many instances, the rating categories can represent degrees 
of quality, with appropriate labelling and description of each step. Care is needed in any quantitative 
analysis of such data. 
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Input measures 

Management preparation and quality 

T Managerial leadership qualities 
T Sufficiency and quality of facilities 
T Extent and strength of contacts with industry 
T Scope and quality of strategic planning 

T Satisfaction of Australian Quality Training Framework requirements for registration of registered 
training organisations 

T Maintenance of Australian Quality Training Framework requirements for registration of registered 
training organisations 

Provision of opportunities for staff development 
T Strategic approach to human resource development 

Source for data: institutional database; management; staff; independent evaluator 

Teacher preparation and quality 

T Scope and quality of domain expertise and industry experience 
T Knowledge of relevant training package requirements 

T Awareness of Australian Quality Training Framework teaching and assessment requirements 
T Extent of staff completion of trainer and assessor qualifications 
T Maintenance of experience, knowledge and qualifications 
Source for data: institutional database; management; staff 

Facilities and support systems 

T Provision of appropriate spaces and equipment for learning 
T Provision of range of learning modes and opportunities 
T Provision of appropriate information and guidance systems 
T Provision for resolving work— study and personal conflicts 
T Flexibility in catering to the diversity of student backgrounds and needs 
T Removal of barriers to participation for equity groups 
T Development of special response mechanisms for different equity groups 
T Representation of student diversity in institutional decision-making processes 
Source for data: institutional database; management; staff; employers, equity groups 

Process measures 

Process measures are mainly subjective and used locally. Three main categories have been identified: 
organisational culture/ climate; organisational capacity for change; and program delivery. Within each of 
these categories, sub-categories have been identified by analytical clustering of common ideas. Finally, 
within each sub-category, potential indicators have been listed. These can be developed later into scales to 
serve as performance measures. 
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Organisational culture/ climate 

Goals 

P Explicit, meaningful, actionable goals 

P Satisfaction of market demand (occupies appropriate niche) 

P Strategic plans and management plans (existence of) 

P Strategic plans and management plans (quality of) 

P Balance between internal and external press 
P Structure for managing goals (existence of) 

P Structure for managing goals (effectiveness of) 

P Organisational participation in identifying goals 
P Alignment of understanding of sectors or departments 
Source for data: management; staff; independent evaluator 

Projected identity and image 

P Identification of and capitalisation on unique characteristics 
P Extent to which the institution is well positioned in the training market 

P Clarity of identity/image (for example, ‘a VET provider’ versus ‘the foremost underwater welding 
training institute’) 

P Community awareness and distinctions from other institutions 
P Attraction to intended clientele (students and employers) 

P Existence of process for reviewing organisational identity 
Source for data: management; staff; students; employers 

Consistency of the message systems 
P Clarity of goals and direction 
P Awareness of the message systems 
P Coherence of the message systems 

P Transparency of organisational decision-making structure 

P Construction of organisational identity 

P Staff involvement versus isolation 

P Feelings of inclusiveness/isolation 

“y* Information flow 

Source for data: management; staff 

Capacity I flexibility to respond to change 
P Sensitivity to need for change (reactive) 

P Sensitivity to need for change (proactive) 

P Existence of competencies to implement change (existence of) 

P Capacity to obtain or generate competencies to implement change 
P Readiness to respond to change (team-building) 
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P Perception of appropriate reward/benefit structure to support change 
P Appropriate anticipation of work demands resulting from change 
P Alignment of systems and goals (for example, technology, accounting, scheduling) 
P Establishment of networks for collaborative activities 
Source for data: staff; employers; independent evaluator 

Staff orientations and perspectives 
P Staff morale 
P Sense of common purpose 
P Pride in the institution 

P Extent to which staff feel their opinions/ contributions are valued 
P Teacher expectations of students (pygmalion effect) 

P Engagement with students (standards/ challenge/broadening of horizons) 

Source for data: staff 

Student experiences 

P Image of the institution (warm, welcoming, supportive) 

P Resonance with the institution (comfortable, pleasant, secure) 

P Staff as role models 
P Sense of purpose/futility 
P Institution generates hope 
P Institution supports aspirations 

P Institution connects to future possibilities/opportunities 
P Self-concept: self-esteem or efficacy 

P Feeling of self-control (internal versus external locus of control) 

P Would recommend institution to others 
Source for data: students 

Interface with industry I community 

P Perceptions of competence, excellence, relevance, morale 
P Responsiveness to and understanding of employer needs 
P Entrepreneurial, engaging, proactive 
P Delivery on expectations 
Source for data: employers 

Organisational capacity for change 

Organisational style and preparedness for change 
P Balance between holistic identity and sectional initiative 
P Existence of clear mechanisms for managing change 
P Existence of explicit planning strategies 
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A Match between rhetoric and reality of change 

-p Extent to which management of change is proactive (internally driven) 

-p Extent to which management of change is reactive (externally driven) 

A Imaginative response to challenges (both internal and external) 

Source for data: management; staff; independent evaluator 

Change management approaches 

Extent to which appropriate consideration is given to: 

dr Readiness for change within the institution 

dr Development of a common vision for the future 

dr Development of political and community support 

dr Staging of transition between old and new 

dr Impact of any sectional change on other internal sections 

dr Impact of change on external public relations 

“y* Impact of change on industry employers 

dr Political or public sector consequences of change 

dr Consequences of change for the broader community 

dr Consequences of change for registration/ accreditation 

Source for data: management; staff; independent evaluator 

Organisational review procedures 

dr Adoption of review strategies for monitoring institutional performance 
dr Adoption of strategies to recognise and reward initiative and effort 
dr Involvement of industry partners in collaborative efforts 
Source for data: management 

Program delivery 

Program quality 

dr Quality of design, content and organisation of (each) program 

dr Implementation of review procedures for assuring program quality 

dr Excellence of teaching, learning support and assessment 

dr Range and adaptability of teaching approaches 

dr Implementation of competency-based training and assessment 

dr Use of external follow-ups (or call-back arrangements) to track job placements 

Source for data: teachers; independent evaluator 

Teacher I trainer quality 

dr Extent of involvement in opportunities for professional development 
dr Extent and depth of reflection, evaluation and change concerning teaching 
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A Extent and depth of adaptive and creative use of training packages 
Development of networks among fellow teachers/trainers 
S Development of networks with industry and community 

Willingness to take the initiative in program design and promotion 
Source for data: teachers; independent evaluator 

Student engagement 

S Adaptation to student expectations and aspirations 
S Presentation of interesting and challenging learning tasks 
A Articulation and coherence between components of (each) program 
S Communication of connections between competencies and workplace demands 
S Communication of future pathways and opportunities. 

Source for data: students; teachers; independent evaluator 

Student support 

S Sensitivity and imagination in dealing with work— study conflicts 
S Provision of information on pathways to work and further study 
S Development of effective information and guidance systems 
S Assistance to students in finding work and personal presentation 
S Identification of students at risk and design of support mechanisms 
Evidence of scaffolding of the learning experiences for students 
Development of ethos of inclusiveness and respect for difference 
Source for data: students; independent evaluator 

Outcome measures 

Student/ trainee outcomes 

S Satisfaction of needs, interests, aspirations of students/ trainees 

Development of self-confidence, generic competencies and lifelong learning skills 
A Satisfaction with resulting employment opportunities and prospects 
A Usefulness of competencies/ qualifications gained to employment prospects 
A Relevance of competencies/ qualifications gained to workplace performance 
A Contribution of competencies/qualifications gained to increased job satisfaction 
A Contribution of competencies/qualifications gained to increased opportunities 
A Contribution of competencies/ qualifications gained to increased earnings 
Source for data: students 
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Employer outcomes 

A Employability and developmental potential of people with training 
A Relevance and currency to the enterprise of competencies gained in training 
A Contribution of competencies gained in training to enterprise activities 
Source for data: employers 



NCVER data sources and their use 

An initial objective of the project was to analyse in detail the national VET data collection to assess the 
extent to which it could used to provide information at the institution level to assist with monitoring 
‘institutional health’ and to assist with self-evaluation and improvement. 

Data are collected nationally from the VET sector in relation to a number of different aspects of their 
performance, and are forwarded to the Data Reporting and Analysis Branch of NCVER for 
incorporation in a national reporting database. Data collections contain information relating to: 

A VET providers 

A VET New Apprenticeships 

A VET financial data. 



VET providers 

The standard for VET providers is comprised of nine data files, shown in table 8. 



Table 8: Files forming the VET providers collection 



File name 


Data collected 


NAT00010 


Training organisation 


NAT00020 


Training provider location 


NAT00030 


Course 


NAT00060 


Module/unit of competency 


NAT00080 


Client 


NAT00090 


Client disability 


NAT001 00 


Client prior educational achievement 


NAT001 20 


Enrolment 


NAT001 30 


Qualification completed 


NAT001 80* 


Unit of competency completed — not uniformly collected 


NAT00200* 


New Apprenticeships — not collected 



VET New Apprenticeships 

The standard describes the New Apprenticeships collections as being comprised of seven files (shown in 
table 9). These files are notionally linked through the client identifier as noted in the provider file. 
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Table 9: Files forming the VET New Apprenticeships collection 



File name 


Data collected 


No. of fields 


% numeric 


% of data 


NAT00010 


Training organisation 


11 


9.0 


7.3 


NAT00080 


Client 


13 


0.0 


0.0 


NAT00100 


Client prior educational achievement 


2 


50.0 


23.1 


NAT00150 


Training contract transaction 


19 


10.5 


4.3 


NAT001 60 


Employer 


6 


12.0 


6.3 


NAT001 70 


Qualification 


4 


1.0 


2.5 


NAT00190 


Registered training organisation 


2 


0.0 


0.0 



VET financial data 

The VET financial data collection relates to VET provision from public funds. The financial data 
holdings represent accrual-based statements associated with audits of key pieces of information and are 
based upon the reports on VET activity of the states, territories and ANTA. 

The four primary financial statements collected are characterised in the standard as: 

T financial performance 

T financial position 

T cash flows 

T notes forming the above. 

VET financial data are reported at the state level and represent revenue to providers. Institutional data do 
not appear to be readily available for analysis. Certainly it appears that a number of issues, including 
current protocols for access to data, would make it difficult to access data on institutional financial 
performance at the institution level. 

Student and employer survey data 

Student Outcomes Survey 

As part of the national data collections, NCVER also conducts a number of surveys. These surveys 
include surveys of both TAFE graduands and TAFE module completers. These surveys collectively form 
the Student Outcomes Survey database. The surveys are intended to capture the attitudes of a 
representative sample of VET clients (students) from a number of settings. 



Table 1 0: Stated content domains of the Student Outcomes Survey 



Graduate questionnaire sections 


Module completer questionnaire 


About you 


About you 


Before starting your course 


About your training 


Six months before the course 


More about you 


About your course of study 


Before you start your training 


Employment during the final semester 


Six months before training 


Your work situation at 25 May 2001 


More about your training 


Further study 


Employment during your training 


Opinions on the course 


Your work situation at 25 May 2001 


Suggestions for improvement 


Your opinions on the training 




Suggestions for improvement. 
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Employer S atisfaction Survey 

The Computer Assisted Telephone Interview is a telephone survey of employer experience and views on 
vocational training. The survey is conducted joindy by ANTA and NCVER. It provides a snapshot of 
employer characteristics, their satisfaction with the VET sector, and with trainees’ acquired skills. In 
addition there is some opportunity for employers to indicate areas for improvement. 

The survey has been conducted biennially since 1995. In the first three collections the survey sample was 
targeted at employers who had at least one employee identified as having recendy completed a VET 
course with a minimum of 200 hours training. The latest survey has broadened the scope to include all 
employers. Consequently, variations in the scope and content domains of the various instruments do not 
permit direct comparison of survey data across all years. 

Relevance of national data to the model 

The range of potential indices outlined in the potential model incorporate, but extend well beyond data 
held by NCVER. However, the student and financial statistical collections form an important base of 
information for a number of potential indices outlined above, and are already used at a state and 
institution level for performance reporting. Performance analysis in relation to apprenticeships and 
traineeships is predominantly at the state and national levels, and institutional performance through this 
specific mode of delivery has not been a primary area of performance measurement to date, possibly 
because apprentice and trainee intakes are influenced by many factors outside the control of VET 
institutions. 

Reporting on student and employer satisfaction also occurs mainly at the state and national level, and in 
the case of employer satisfaction, tends to provide information on general employer attitudes to training 
institutions rather than to a specific institution. Thus there are major limitations on the use of these data 
for performance monitoring at the institution level, and many VET providers already carry out their own 
client surveys for this information. There is also a requirement for immediacy and for information on 
client satisfaction at the individual teaching department level, neither of which is available from national 
data reporting. 

As such, NCVER student and finance data collections, based as they are on national standards and subject 
to a national quality assurance process, are an important source of base information for institutional 
performance monitoring. However, they constitute only some elements of the data required to support 
the indices in the model outlined above. 



Implementing the model 

What has been offered here is a comprehensive mapping of potential performance indicators/ measures 
pertinent to VET institutions’ undertaking self-monitoring of institutional ‘health’ and effectiveness. The 
listing of indicators/measures given is more extensive and systematic than any current lists for this 
purpose and have been distilled from existing research and development relating to institutional 
functioning in vocational education and training and more generally. These indicators/ measures and the 
framework within which they have been developed offer a new way of thinking about monitoring 
institutional health and effectiveness, with a view to improvement. 

The sets of input and output/ outcome measures offered here are somewhat different from the existing 
ones adopted at a national level, but are not conceptually different from those proposed for particular 
equity groups, such as Aboriginal and Torres Strait Islanders. This is a case where thinking through the 
necessary provisions for effective equity can lead to considerations which have more general application. 

The feature which most marks this set of indicators/measures as very different from existing ones is the 
substantial emphasis on process or enabling aspects of institutional health and effectiveness (such as 
quality of decision-making and institutional climate and culture). This is a necessary outcome of 
developing a broader conceptualisation of what constitutes institutional health and effectiveness and what 
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mechanisms can be used to change and improve the situation. From the perspective offered here, these 
process indicators /measures indicate intermediate outcomes — in the sense that they are desirable in 
themselves and are consequences of decisions about how the resources of the institution are to be 
deployed and how program delivery is to be managed — and also indicate mediating inputs for reaching 
the end goals of the institution. That is, they are important in their own right as intermediate goals as well 
as being the potential agency for reaching the ‘ultimate’ or ‘key’ goals (expressed in terms of outputs/ 
outcomes) . 

It is necessary to say ‘potential agency’ here because not all of the intermediate goals can be shown to 
contribute unequivocally to ultimate goals. Nevertheless, they may still be valued in their own right. Thus, 
for example, it is desirable for student/ trainee experiences to be pleasant and helpful, irrespective of 
whether they contribute directly and of themselves to the ultimate goal of the development of 
appropriate competencies. The affective response of students/trainees to their training may influence 
their learning (in combination with other factors), but even if it did not, we would still want the response 
to be positive — as a desirable value itself. Of course, the development of appropriate competencies is still 
a key touchstone of institutional effectiveness, without which any amount of positive feelings are 
irrelevant. 



Strategy for use 

There is, of course, no suggestion that VET institutions should or could collect information on all of 
these indicators/ measures on a regular and ongoing basis. Rather, they represent a map of possibilities 
from which selection could be made for particular circumstances. Over some time, perhaps five years, 
comprehensive coverage could be expected, such that all factors and their implications have been 
considered on a planned strategic cycle. This could be implemented in the form of a rolling series of 
action learning plans. Each of these would involve a systematic process whereby defining aspects of the 
organisation are evaluated, collecting data pertinent to those aspects, interpreting the data, drawing 
implications for action and implementing those actions, followed by repetition of the whole process. Each 
action learning plan would overlap with other action learning plans at various phases of their 
development. Different parts of the organisation could be involved in different action learning plans and 
at different stages of their implementation, although there also might be one plan which looks at the 
organisation as a whole rather than separate parts or programs. It has not been part of this project to 
work out these kinds of strategies, but there are plenty of seminal ideas in the literature on action learning 
and organisational evaluation. 

It is also necessary to operationalise the indicators/ measures through the development of appropriate 
scales and appropriate data collection procedures. Ideally, scales should be developed to tailor the features 
of the indicator /measure so that the scale points are defined to represent explicit standards along a 
continuum. It is possible to use simple qualitative ratings; for example, of quality (‘poor’, ‘satisfactory’, 
‘good’, ‘excellent’) or satisfaction (‘very dissatisfied’, ‘dissatisfied’, ‘satisfied’, ‘very satisfied’) or extent (‘not 
at all’, ‘to a little extent’, ‘to some extent’, ‘to a great extent’, ‘to a very great extent’, ‘completely’). 

However, scale points may be better represented by descriptive statements which indicate the operational 
characteristics being referenced. 

Pointers and assistance can be found in the many books and resources available on organisational 
improvement and evaluation. One important issue for resolution is whether the evaluation process is best 
handled internally as an incidental part of organisational management, or externally through the services 
of a contracted consultant. There is a strong move towards internal evaluation, but this can involve the 
services of external consultants (Love 1991). Different approaches are needed for different information 
needs, with different information attributes taking on different characters, depending on whether they 
service strategic, managerial or operational decisions (Love 1991). Love (1991) suggests ten information 
attributes for consideration: type of question (what if versus what is); time horizon; data source; data 
characteristics; level of detail; level of analysis; frequency of reporting; scope of reporting; accuracy of 
reporting; and mode of reporting (p.28). 
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Chang and De Young (1995) provide simple and practical advice on how to manage organisational 
evaluation. They offer a ‘measurement linkage model’, involving a way to link all work group assessment 
systems through defined key result areas and key indicators which ‘cascade down through the 
organisation’ (p.14). On a more conceptual level, Trice Gray and Associates (1998) propose an inclusive 
and ongoing process somewhat akin to traditional action learning. Their three basic steps are: ask good 
questions; collect information; and share and use the information to make good decisions. This 
‘permeating and participative culture’ of organisational evaluation and improvement is also akin to the 
creation of a ‘learning organisation’. There are many useful resources for developing learning 
organisations and these may provide a useful line of further development beyond this project (see Clarke 
2001; Senge 1990; Senge et al. 1994; Senge et al. 1999; Senge et al. 2000). 

A collaborative approach to the further development of the proposed model (and others which may 
emerge) would reduce costs, support benchmarking and allow for outcomes of model development and 
learning from the developmental process, to be shared. 
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